DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/29/2026 has been entered.
Response to Arguments
Applicant’s arguments, see Remarks page 10, filed 06/29/2026, with respect to the rejections of claim(s) 10-11 and 15-16 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejections of claims 10-11 and 15-16 have been withdrawn.
Applicant’s arguments, see Remarks page 10-14, filed 06/29/2026, with respect to the rejections of amended claim(s) 1, 12, and 17 under 35 U.S.C. 103 have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kargieman et al. (US2017250751A1) hereinafter referenced as Kargieman, in view of Keisler et al. (Visual search over billions of aerial and satellite images) hereinafter referenced as Keisler, and Imig et al. (US2023196201A1) hereinafter referenced as Imig.
Regarding claim 1, Kargieman discloses: An apparatus, comprising: a first device in a constrained-environment device, the first device including at least one memory having processor-executable code stored therein, and at least one processor that is adapted to execute the processor-executable code, wherein the processor-executable code includes processor-executable instructions that, in response to execution, enable the first device to perform actions (Kargieman: Figure 1; 0021-0022: “As illustrated in FIG. 1, a system 100 includes a satellite 102. The satellite 102 includes a payload 104, which in many cases will include an imaging system 106 . The imaging system may include one or more cameras (also referred herein as “imaging sensor”), such as a macro camera 108 designed to capture relatively wide field of view imaging data, and a micro camera 110 having a relatively narrow field of view and which may generally also provide more detailed resolution images…A control system 118 may be comprised of one or more on board computers for handling the functions of the payload 104 and other systems on board the satellite. The control system 18 may include one or more processors 120 and computer-readable media 122.”), including:
storing image metadata and raw image data obtained from sensors on the constrained-environment device, wherein the raw image data includes a plurality of images, and wherein the image metadata includes, for each image of the plurality of images, a geographic location associated with the image (Kargieman: 0023: “The satellite system 100 allows the satellite 102 to do in real-time much of the image processing pipeline, including transforming raw sensor data into geo-located, normalized, ortho-rectified images that are ready for interpretation and/or analysis. Much of this ability is stored as instructions, processes, and logic within the computer-readable media 122 of the satellite 102.”;
0031: “ The imaging module 132 is primarily responsible for sending instructions to the hardware components of the imaging system 106 for capturing image data. The imaging module 132 may further include instructions for collecting, storing, cataloging, timestamping collected images, determining exposure, amplification gains, and image statistics, among other things. Furthermore, the imaging module 132 may further include instructions for processing the image data, such as segmenting, stitching, correcting, or rectifying the image data.”);
receiving a list of geographic areas of interest (Kargieman: 0052: “a customer requiring specific satellite imaging data may input a request regarding an area of interest (AOI) or a point of interest (POI), a particular type of AOI as a polygon on the surface of the Earth in a particular coordinate system. Associated with the AOI, the customer selects and fills specific metadata to specify a Service Level Agreement (SLA). The SLA can include…sensor processing pipeline specifications (for radiometric correction, ortho-rectification, stitching, projection models, sensor models, atmospheric correction, geo-location, etc.); sensor analysis workflow (including raster and vector processing blocks, machine learning modules, intermediate storage procedures, etc.); data required on the ground (data types, results of sensor analysis workflow, and frequencies, or trigger conditions [for example, only download X if a certain parameter exceeds a threshold])”;
0136: “The modules allow the satellite system to receive, from a ground-based device or from another satellite system, a request for data that includes a request to determine one or more parameters relating to at least one object within a particular geographic region. Based upon the request, the satellite system captures, by a first sensor of the one or more sensors, one or more images associated with the particular geographic region. The modules further allow the satellite system to analyze the one more images to determine that a first subset of the images provides an unobstructed view of the particular geographic region. This may be based, for example, on determining that there is little or no cloud cover, such that the geographic region is visible in the images. Other forms of obstruction may be detected, such as fog, shadow, smoke, or other such conditions that may inhibit a clear view of the particular geographic region.”);
for each embedding of a plurality of embeddings corresponding to an image having a geographic location proximate a geographic area of interest on the list, performing model orchestration on the embedding to select a model based at least on the geographic location of the image corresponding to the embedding (Kargieman: 0007: “if the satellites themselves can employ algorithms that detect ship candidates, such as through edge detection or object detection, then a significantly smaller number of images that contain ship-candidates may be downloaded to a ground-based station. Furthermore, the satellite may not necessarily have to capture images of the entire Gulf of Mexico, especially when the satellite is tasked to a small geo-located area around known ship candidates' coordinates.”;
0052-0053: “Associated with the AOI, the customer selects and fills specific metadata to specify a Service Level Agreement (SLA). The SLA can include…sensor processing pipeline specifications (for radiometric correction, ortho-rectification, stitching, projection models, sensor models, atmospheric correction, geo-location, etc.); sensor analysis workflow (including raster and vector processing blocks, machine learning modules, intermediate storage procedures, etc.); data required on the ground (data types, results of sensor analysis workflow, and frequencies, or trigger conditions [for example, only download X if a certain parameter exceeds a threshold])…the customer is given the ability to specify (or provide) programs to be uploaded to the satellite to be executed, and some of these may be executed on-demand. Examples of this type of service, for example, are to allow a customer to train and upload their own machine learning algorithms, which can be configured to conduct image processing and analysis”);
performing analysis on an image tile corresponding to the embedding using the selected model to determine whether the image tile includes a target object; selecting a subset of embeddings from the plurality of embeddings that correspond to a subset of image tiles from a plurality of image tiles that include the target object (Kargieman: 0073: “areas of interest (AOI) may be projected as vector layers onto the ortho-rectified image mosaics, and used a basis for selecting or discarding data for further processing, or for downloading to the ground. In other words, an AOI may be segmented from a larger image and only the segmented portion of the image data is saved and/or analyzed while the remainder of the image data is discarded. Segmentation may be performed, as an example, by drawing a polygon around pixels that exhibit the features of interest, and only saving the pixels found within the polygon and discarding or compressing more aggressively the remainder of the imaging data.”);
and during a downlink session from the constrained-environment device, downlinking image tiles from among the subset of image tiles (Kargieman: 0007: “if the satellites themselves can employ algorithms that detect ship candidates, such as through edge detection or object detection, then a significantly smaller number of images that contain ship-candidates may be downloaded to a ground-based station. Furthermore, the satellite may not necessarily have to capture images of the entire Gulf of Mexico, especially when the satellite is tasked to a small geo-located area around known ship candidates' coordinates.”;
0073: “areas of interest (AOI) may be projected as vector layers onto the ortho-rectified image mosaics, and used a basis for selecting or discarding data for further processing, or for downloading to the ground. In other words, an AOI may be segmented from a larger image and only the segmented portion of the image data is saved and/or analyzed while the remainder of the image data is discarded. Segmentation may be performed, as an example, by drawing a polygon around pixels that exhibit the features of interest, and only saving the pixels found within the polygon and discarding or compressing more aggressively the remainder of the imaging data.”;
0081: “By pushing cloud-detection algorithms to the satellites, and having the ability to segment the images in-orbit and only download the parts of the images that are cloudless, we realize an immediate 60% direct gain in the information we can bring to the ground with the same downlink system. ”;
Wherein only segments/tiles of images containing a target object are downlinked).
Kargieman does not disclose expressly: providing a plurality of images tiles such that the plurality of images tiles includes, for each image of the plurality of images, evenly-spaced portions of the image; via an embedding-generation model, generating a plurality of embeddings based on the plurality of image tiles.
Keisler discloses: A system for performing a visual search of aerial and satellite image by converting images into feature vectors and comparing their similarities. (Keisler: Abstract), Wherein the system further comprises:
providing a plurality of images tiles such that the plurality of images tiles includes, for each image of a plurality of images, evenly-spaced portions of the image (Keisler: 2. Imagery: “We chunked this imagery into square image tiles, 128 pixels on a side. Adjacent tiles overlap by 64 pixels in the vertical and horizontal directions.”);
via an embedding-generation model, generating a plurality of embeddings based on the plurality of image tiles (Keisler: 3.2. Customizing for Aerial and Satellite Imagery: “For Landsat 8, the OSM classes were less useful due to the much coarser spatial resolution, so we took an unsupervised approach instead. Specifically we used an autoencoder to compress the 2048 features from the penultimate layer of ResNet50 into 512 binary features using a 2048-1024-512-1024-2048 autoencoder architecture. The autoencoder was trained by passing Landsat 8 imagery through the original ResNet50 network.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to perform the generation of a plurality of image tiles embeddings as taught by Keisler using the satellite images disclosed by Kargieman. The suggestion/motivation for doing so would have been “We chunked this imagery into square image tiles, 128 pixels on a side. Adjacent tiles overlap by 64 pixels in the vertical and horizontal directions. While this overlapping tiling scheme requires two times more data than a non-overlapping scheme, it gives any object the chance to be roughly centered in at least one image tile. We used the Descartes Labs Platform7 to access and process this imagery. This platform allows the user to search for the existence of imagery, quickly load that imagery into memory in the form of a python ndarray, and parallelize computations over tens of thousands of CPUs” (Keisler: 2. Imagery; Wherein the image tiling allows for large images to segmented and processed in parallel, allowing for compact features and quick processing.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Kargieman in view of Keisler does not disclose expressly: via the at least one processor, performing prioritization of image tiles among the subset of image tiles using the subset of embeddings; and during a downlink session from the constrained-environment device, downlinking image tiles from among the subset of image tiles based, at least in part, on the prioritization.
Keisler further discloses: performing prioritization of image tiles among a subset of image tiles using a subset of embeddings (Keisler: 4.1. Direct Search: “A query image tile is selected. We retrieve the binary feature vector for this image tile using Redis. We compute the Hamming distance between the query vector and all vectors using a pre-computed binary file that contains all binary feature vectors…We keep the k-nearest feature vectors and determine their corresponding images tiles using a second index file.”; Wherein the image tiles corresponding to the k-nearest embeddings to the query embedding are prioritized and retrieved.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to prioritize the selected subset of image tiles for downlinking disclosed by Kargieman in view of Keisler using the image tile prioritization method further taught by Keisler. The suggestion/motivation for doing so would have been “We first define a distance between vectors. We use the Hamming distance: the number of bits that differ between the two binary vectors. A small distance is associated with visual similarity. Next we need to find the k-nearest vectors to a query vector.” (Keisler: 4. Search; Wherein the feature vector similarity searching allows for retrieval/prioritization of image tiles containing a desired object, based on the tiles’ visual similarities.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Kargieman in view of Keisler does not disclose expressly: for each embedding of the plurality of embeddings corresponding to an image having a geographic location proximate a geographic area of interest on the list, performing model orchestration on the embedding to select a model from a plurality of different models based at least on the geographic location of the image corresponding to the embedding.
Imig discloses: for each embedding corresponding to an image, performing model orchestration on the embedding to select a model from a plurality of different models based at least on the geographic location of the image corresponding to the embedding (Imig: 0028: “AI inference platform (AIP) orchestrates between the input sensor data and output feeds…”;
0036: “the AIP 110 is configured to change the one or more models 130 automatically or dynamically based on the processed sensor data. As an example, the AIP 110 is configured to change the one or more models 130 in response to changes to the sensor data collected via the one or more sensors 120.”;
0051: “the AIP is configured to process the video input using three models in serial so that they can actually feed off of each other. In some examples, downstream processors and/or models can utilize upstream results within the processing logic. For example, a cloud detection algorithm runs first on a satellite image, then a ship detection runs next and only runs in places where clouds were not detected, improving efficiency and speed.”;
0052: “the AIP can be used in a real time-oriented system. In some examples, each of the one or more processors have a specific budget. For example, the AIP is configured to send a video to a model for processing the video and receive the first processing results in 200 milliseconds. In one example, the AIP can output the first processing results to another model (e.g., geo-registration processor) with a specific budget of 500 milliseconds because it's doing more processing (e.g., more intense processing) in the computer vision process, and after receiving the result, the AIP may send the result out. In one example, the AIP can send the first processing results in parallel to a plurality of models (e.g., AI models, object detection algorithms, etc.), for example, so the plurality of models can benefit from the first processing results (e.g., the upstream processing). In certain embodiments, the AIP includes application programming interface (API) to send one or more inputs to and receive one or more results from one or more processors.”
0059-0060: “In some examples, the AIP is configured to coordinate and/or synchronize a first sensor data (e.g., a video) with a second sensor data (e.g., GPS coordinates, telemetry data). For example, the second sensor data provides geospatial location information of the first sensor data…According to certain embodiments, the AIP is configured to link the second sensor data (e.g., GPS coordinates) to the first sensor data (e.g., a video). For example, the AIP links some geospatial data to a video to actually get a sense of what a user is actually looking at.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to perform the model orchestration disclosed by Kargieman in view of Keisler using a plurality of different models as taught by Imig. The suggestion/motivation for doing so would have been “In one example, the AIP is configured to process the video input using two models in parallel. In one example, the AIP is configured to process the video input using three models in serial so that they can actually feed off of each other. In some examples, downstream processors and/or models can utilize upstream results within the processing logic. For example, a cloud detection algorithm runs first on a satellite image, then a ship detection runs next and only runs in places where clouds were not detected, improving efficiency and speed…For example, the AIP is configured to send a video to a model for processing the video and receive the first processing results in 200 milliseconds. In one example, the AIP can output the first processing results to another model (e.g., geo-registration processor) with a specific budget of 500 milliseconds because it's doing more processing (e.g., more intense processing) in the computer vision process” (Imig: 0051-0052; Wherein model orchestration is adjusted in order to improve processing efficiency and speed.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Kargieman in view of Keisler with Imig to obtain the invention as specified in claim 1.
Regarding claim 2, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein performing the prioritization of the image tiles is further based on the image metadata (Keisler: 1. Introduction: “What is lacking is metadata encoding a deeper visual or semantic understanding of the image content. Ideally, this “visual metadata” …would encode some meaningful notion of “distance” between images. The feature vectors presented in this paper provide a step in that direction.”; 4.1. Direct Search: “We compute the Hamming distance between the query vector and all vectors using a pre-computed binary file that contains all binary feature vectors…We keep the k-nearest feature vectors and determine their corresponding images tiles using a second index file.”; Wherein the image tile feature vectors encode metadata extracted from the image tiles.).
Regarding claim 3, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein the constrained-environment device is at least one of: an Internet of Things device that is in a constrained environment, an orbiting satellite, a spacecraft, or a stationary platform (Kargieman: Figure 1; Abstract).
Regarding claim 4, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein the sensors on the constrained-environment device include at least one of a camera, a synthetic aperture radar, a thermal imaging sensor, a hyperspectral sensor, or a video sensor (Kargieman: Figure 1; 0021: “As illustrated in FIG. 1, a system 100 includes a satellite 102. The satellite 102 includes a payload 104, which in many cases will include an imaging system 106. The imaging system may include one or more cameras (also referred herein as “imaging sensor”), such as a macro camera 108 designed to capture relatively wide field of view imaging data, and a micro camera 110 having a relatively narrow field of view and which may generally also provide more detailed resolution images. A general purpose camera 112 may be incorporated for broad field of view imagery and may provide imaging data input to a cloud detection algorithm. A TIR camera 114 may also be incorporated for thermal imaging data and infrared imaging data. Additional cameras may include a multispectral camera or a hyperspectral camera, among others.”).
Regarding claim 5, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein the embedding-generation model includes at least one of an unsupervised representation learning model, a self-supervised representation learning technique, or a supervised representation learning technique (Keisler: 3.2 Customizing for Aerial and Satellite Imagery: “For Landsat 8, the OSM classes were less useful due to the much coarser spatial resolution, so we took an unsupervised approach instead. Specifically we used an autoencoder to compress the 2048 features from the penultimate layer of ResNet50 into 512 binary features using a 2048-1024-512-1024-2048 autoencoder architecture. The autoencoder was trained by passing Landsat 8 imagery through the original ResNet50 network.”).
Regarding claim 6, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein the embeddings are feature vectors of floating-point numbers (Keisler: 3.1. Binary Features: “To that end, we encouraged the network to make floating-point features very close to 0.0 or 1.0 at the layer of interest by injecting noise with an amplitude comparable to the width of the layer’s activation function (Salakhutdinov and Hinton, 2009).”).
Regarding claim 7, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein the embeddings are feature vectors each having at least 256 dimensions (Keisler: 3.2 Customizing for Aerial and Satellite Imagery: “For Landsat 8, the OSM classes were less useful due to the much coarser spatial resolution, so we took an unsupervised approach instead. Specifically we used an autoencoder to compress the 2048 features from the penultimate layer of ResNet50 into 512 binary features using a 2048-1024-512-1024-2048 autoencoder architecture.”).
Regarding claim 8, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein the constrained-environment device is an orbiting satellite (Kargieman: Figure 1; Abstract).
Regarding claim 9, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 8, wherein the plurality of images includes a plurality of satellite images (Kargieman: Abstract: “the satellite is able to make decisions about capturing imagery data, including image data processing, object detection, image segmentation, and re-orientation of the satellite based upon the opportunity, from the satellite's perspective, for capturing image data of areas or objects of interest. Through the use of machine learning and in-orbit image analysis, the satellite may transmit only a subset of the captured images, portions of the captured images, or the result of image analysis, thereby efficiently using the downlink communication channel.”).
Regarding claim 10, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 1, wherein using the subset of embeddings to perform the prioritization of image tiles among the subset of image tiles includes comparing embeddings in the subset of embeddings to embeddings in a set of target embeddings (Keisler: 4.1. Direct Search: “A query image tile is selected. We retrieve the binary feature vector for this image tile using Redis. We compute the Hamming distance between the query vector and all vectors using a pre-computed binary file that contains all binary feature vectors.”; Wherein the query image tile/vector constitutes a target embedding).
Regarding claim 11, Kargieman in view of Keisler and Imig discloses: The apparatus of claim 10, wherein comparing the embeddings in the subset of embeddings to the embeddings in the set of target embeddings includes determining which embeddings in the subset of embeddings are close, in a vector space, to the embeddings in the set of target embeddings (Keisler: 4.1. Direct Search: “A query image tile is selected. We retrieve the binary feature vector for this image tile using Redis. We compute the Hamming distance between the query vector and all vectors using a pre-computed binary file that contains all binary feature vectors…We keep the k-nearest feature vectors and determine their corresponding images tiles using a second index file.”).
As per claim(s) 12, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 13, arguments made in rejecting claim(s) 8 are analogous.
As per claim(s) 14, arguments made in rejecting claim(s) 5 are analogous.
As per claim(s) 15, arguments made in rejecting claim(s) 10 are analogous.
As per claim(s) 16, arguments made in rejecting claim(s) 11 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 1 are analogous. In addition, paragraph 0024 of Kargieman discloses a “processor-readable storage medium, having stored thereon processor- executable code”.
As per claim(s) 18, arguments made in rejecting claim(s) 8 are analogous.
As per claim(s) 19, arguments made in rejecting claim(s) 5 are analogous.
As per claim(s) 20, arguments made in rejecting claim(s) 10 are analogous.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
