DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-23 is/are rejected under 35 U.S.C. 103 as obvious over Brewer et al (Brewer, "Granularity at Scale: Estimating Neighborhood Socioeconomic Indicators From High-Resolution Orthographic Imagery and Hybrid Learning", 21 February 2024, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 17, pp. 5668-5679, hereafter referred to as Brewer) in view of Mnih et al (Mnih, "Recurrent Models of Visual Attention", 2014, Advances in neural information processing systems, 27, hereafter referred to as Mnih), further in view of Park et al (Park, "Learning economic indicators by aggregating multi-level geospatial information.", 2022, Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. No. 11., pp. 12053-12061, hereafter referred to as Park), further in view of Ayush et al (Ayush, "Efficient poverty mapping from high resolution remote sensing images", 2021, Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, No. 1, pp. 12-20, hereafter referred to as Ayush).
Claim 1
Regarding Claim 1, Brewer teaches A method, comprising:
by an interactive neural network inclusive of a first network trained with ImageNet (Brewer in page 5672 left column discloses “The ResNet50-based architecture of Fig. 5 has its base model pre-trained on ImageNet.”), and a third network trained with census data collected for a geographic region (Brewer in Abstract and Section III discloses a neural network pipeline comprising ResNet50 pre-trained on ImageNet for feature extraction from satellite/aerial imagery, with fully connected layers trained using American Community Survey (ACS) census data for a geographic region to predict population density, median household income, etc.),
a) obtaining a satellite image of the geographic region (Brewer in Fig. 2 to Fig.4 and page 5670 right column discloses obtaining aerial imagery of geographic regions);
b) cropping, by a processor coupled to the interactive neural network, the satellite image to define a cropped satellite image inclusive of a selected region of the geographic region wherein each pixel of the cropped satellite image has a brightness value (Brewer in page 5671 discloses cropping/patching satellite imagery to define image patches corresponding to selected geographic regions using TIGER/line shapefiles, wherein each pixel inherently possesses brightness values);
d) sampling, by the processor, a first portion of the brightened satellite image wherein the first portion has a centroid defined by a location in the selected region (Brewer in Fig 3. and Fig. 4 discloses sampling image portions (patches) from satellite imagery corresponding to geographic regions, with each patch centered on a centroid of a Census block group);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Brewer in Abstract and page 5672 left column discloses a “ResNet50-based architecture” is used to extract features from image portions, where “features extracted from the imagery can accurately estimate the density” of neighborhoods. The ResNet50 processes image patches to generate feature outputs indicative of features in the imagery);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Brewer in Section IV.A. discloses that “seven fully connected layers are added after the base model to gradually scale down the feature space to a single output estimation”, where the output estimation is one of “density, MHI, education” at the U.S. Census block level. The fully connected layers process the CNN feature outputs to generate an estimated census metric);
h) comparing, by the second network, the estimated census metric with a corresponding metric from the census data to generate a difference there between (Brewer in Section IV.A. discloses that the model is trained using “L1 loss [i.e., mean absolute error, (MAE)]” where “models are trained until there is no improvement in validation accuracy after five epochs”. The L1 function computes the absolute difference between the predicted census metric and the actual census data metric).
Brewer does not explicitly teach all of a second network comprising a recurrent neural network; c) increasing, by the processor, the brightness value for each pixel in the cropped satellite image to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion; g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold;
However, Mnih teaches a second network comprising a recurrent neural network (Mnih in Abstract discloses “a novel recurrent neural network model”);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Mnih in Abstract discloses a model that processes “selected regions at high resolution” where “the proposed model has a degree of translation invariance built-in, but the amount of computation it per forms can be controlled independently of the input image size”. The model processes selected image regions to generate feature outputs);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Mnih in Abstract discloses that the RNN model extracts “information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution”, where the sequential processing generates task-specific outputs);
i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold (Mnih in Abstract discloses a model that “adaptively select[s] a sequence of regions or locations” and “can be trained using reinforcement learning methods to learn task-specific policies”; the RNN adaptively selects a sequence of image regions, maintaining state across selections).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer by incorporating a recurrent visual-attention network that adaptively selects a sequence of image regions or locations and processes the selected regions at high resolution that is taught by Mnih, since both reference are analogous art in the field of neural-network-based computer vision and image analysis; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer’s satellite-image census-metric prediction with Mnih’s recurrent attention/location-selection model yields the predictable result of adaptively selecting and revising image regions for high-resolution feature processing, thereby reducing unnecessary computation and maintaining task-specific prediction performance.
Brewer in view of Mnih does not explicitly teach all of c) increasing, by the processor, the brightness value for each pixel in the cropped satellite image to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion; g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold.
However, Park teaches e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions (Park in Abstract discloses a deep learning model that predicts economic indicators by “aggregating traits observed from multiple levels of geographical units,” wherein “these images display detailed terrain over large areas and allowing zooming into smaller neighborhoods”; Park’s multi-level geographic aggregation teaches processing satellite imagery at different spatial resolutions (zoom levels) centered on the same geographic area for economic indicator prediction.);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Park in Abstract discloses a “deep learning model” that processes imagery at “multiple levels of geographical units” to extract features, where “the model first measures hyperlocal economy over small communities” and “the next step extracts district-level features by summarizing interconnection among hyperlocal economies”. Each level generates feature outputs indictive of features at that geographical resolution);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Park in ¶ discloses that “the model estimates economic indicators of districts via aggregating the hyperlocal and district information”, where “our new multi-level learning model substantially outperforms strong baselines in predicting key indicators such as population, purchasing power, and energy consumption”. Multi-level features are aggregated to generate estimated economic metrics).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer in view of Mnih by generating and processing satellite-image portions at multiple geographic levels/resolutions that is taught by Park, since both reference are analogous art in the field of geospatial/satellite-image analysis for estimating census indicators; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer in view of Mnih’s adaptive image-region processing with Park’s multi-level geospatial aggregation yields the predictable result of generating feature outputs from different geographic resolutions and aggregating those outputs, thereby improving census/economic metric estimation by capturing both local and broader-area context.
Brewer in view of Mnih, further in view of Park does not explicitly teach all of c) increasing, by the processor, the brightness value for each pixel in the cropped satellite image to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold.
However, Ayush teaches e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions (Ayush in Abstract discloses “a reinforcement learning approach in which free low resolution imagery is used to dynamically identify where to acquire costly high-resolution images”, where the RL agent selects the same spatial location for both low-resolution analysis and high-resolution acquisition);
i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold (Ayush in Abstract discloses “a reinforcement learning approach in which free low resolution imagery is used to dynamically identify where to acquire costly high-resolution images, prior to performing a deep learning task on the high-resolution images”; RL-based dynamic spatial location selection for poverty prediction, where the agent iteratively identifies optimal sampling locations based on prediction performance).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer in view of Mnih, further in view of Park by incorporating a reinforcement-learning approach to dynamically identify where to acquire or process high-resolution images that is taught by Ayush, since both reference are analogous art in the field of geospatial/satellite-image analysis for estimating census indicators; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer in view of Mnih, further in view of Park’s geospatial prediction framework with Ayush’s dynamic high-resolution location-selection approach yields the predictable result of moving/revising a selected centroid or sampling location to obtain more informative high-resolution image portions, thereby reducing acquisition/computation cost while maintaining or improving prediction performance.
Brewer in view of Mnih, further in view of Park, further in view of Ayush does not explicitly teach all of c) increasing, by the processor, the brightness value for each pixel in the cropped satellite image to generate a brightened satellite image.
However, the Examiner takes Official Notice. For “c) increasing, by the processor, the brightness value for each pixel in the cropped satellite image to generate a brightened satellite image”, the Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include image brightness enhancement. The Examiner takes Official Notice that image brightness enhancement was a well-known, standard preprocessing technique in computer vision. A PHOSITA would have recognized that satellite image data is commonly compressed in terms of dynamic range, and that increasing brightness is a predictable preprocessing step.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 2
Regarding Claim 2, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the geographic region has a governing body associated therewith (Brewer in page 5670 discloses “Census data for the United States are acquired through the ACS. Every year, the U.S. Census Bureau contacts approximately 3.5 million households (1 in 40 total households) across the country to participate in the ACS”), and
wherein the census data is collected by the governing body (Brewer in page 5670 discloses “Census data for the United States are acquired through the ACS. Every year, the U.S. Census Bureau contacts approximately 3.5 million households (1 in 40 total households) across the country to participate in the ACS”).
Claim 3
Regarding Claim 3, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the selected region is selected from the group consisting of at least one of a state, a province, a city, a county, and a municipality (Brewer in page 5669 left column discloses “94 of the 100 largest U.S. cities by gross domestic product (GDP)”).
Claim 4
Regarding Claim 4, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the brightness value for each pixel is increased by a factor of at least 2 (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include image brightness enhancement. The Examiner takes Official Notice that image brightness enhancement was a well-known, standard preprocessing technique in computer vision. A PHOSITA would have recognized that satellite image data is commonly compressed in terms of dynamic range, and that increasing brightness is a predictable preprocessing step. The specific multiplication factor of at least 2 is a predicable parameter selection within the range of standard brightness enhancement techniques).
Claim 5
Regarding Claim 5, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the second portion is in a range of 50% to 80% smaller than the first portion (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include parameter selection for multi-resolution sampling. The Examiner takes Official Notice that multi-resolution sampling at different scales was a well-known, standard technique in computer vision, and that a 50-80% reduction in area is a predictable scale range for capturing coarser-resolution contextual features alongside finer-resolution detail. A PHOSITA would have recognized that this range represents a conventional trade-off between spatial coverage and feature resolution in multi-scale image analysis.).
Claim 6
Regarding Claim 6, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the second portion is approximately 75% smaller than the first portion (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include parameter selection for multi-resolution sampling. The Examiner takes Official Notice that multi-resolution sampling at different scales was a well-known, standard technique in computer vision, and that a 75% reduction in area is a predictable scale range for capturing coarser-resolution contextual features alongside finer-resolution detail. A PHOSITA would have recognized that this range represents a conventional trade-off between spatial coverage and feature resolution in multi-scale image analysis.).
Claim 7
Regarding Claim 7, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the centroid comprises a latitude and longitude in the selected region (Brewer in page 5671 discloses geographic boundaries from TIGER archive shapefiles, which define neighborhoods by latitude/longitude polygon coordinates; the centroid of each polygon is a latitude/longitude point).
Claim 8
Regarding Claim 8, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the step of moving includes applying a Gaussian distribution function to govern a distance between the location of the centroid and the revised location (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to apply a Gaussian distribution function to govern the distance between the current centroid location and the revised centroid location. The Examiner takes Official Notice that the Gaussian distribution was a well-known, standard mathematical mechanism for controlling step size in iterative optimization algorithms. A PHOSITA would have recognized that a Gaussian-distributed step size provides smooth, bounded movement that converges predictably, and that applying it to govern aspatial displacement in an iterative refinement process is a predictable engineering choice.).
Claim 9
Regarding Claim 9, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the first output and the second output comprise vector outputs (Brewer in page 5673 left column discloses that ResNet50 generates feature vectors, “2048-D representation” per patch in the semi-supervised approach, and feature space outputs in the supervised approach).
Claim 10
Regarding Claim 10, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 1, wherein the third network comprises a fully connected layer (Brewer in page 5672 left column discloses “seven fully connected layers are added after the base model”).
Claim 11
Regarding Claim 11, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches A method, comprising:
by an interactive neural network inclusive of a first network trained with ImageNet (Brewer in page 5672 left column discloses “The ResNet50-based architecture of Fig. 5 has its base model pre-trained on ImageNet.”),
a) obtaining a satellite image of the geographic region (Brewer in Fig. 2 to Fig.4 and page 5670 right column discloses obtaining aerial imagery of geographic regions);
b) cropping, by a processor coupled to the interactive neural network, the satellite image to define a cropped satellite image inclusive of a selected region of the geographic region wherein each pixel of the cropped satellite image has a brightness value (Brewer in page 5671 discloses cropping/patching satellite imagery to define image patches corresponding to selected geographic regions using TIGER/line shapefiles, wherein each pixel inherently possesses brightness values);
d) sampling, by the processor, a first portion of the brightened satellite image wherein the first portion has a centroid defined by a location in the selected region (Brewer in Fig 3. and Fig. 4 discloses sampling image portions (patches) from satellite imagery corresponding to geographic regions, with each patch centered on a centroid of a Census block group);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Brewer in Abstract and page 5672 left column discloses a “ResNet50-based architecture” is used to extract features from image portions, where “features extracted from the imagery can accurately estimate the density” of neighborhoods. The ResNet50 processes image patches to generate feature outputs indicative of features in the imagery);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Brewer in Section IV.A. discloses that “seven fully connected layers are added after the base model to gradually scale down the feature space to a single output estimation”, where the output estimation is one of “density, MHI, education” at the U.S. Census block level. The fully connected layers process the CNN feature outputs to generate an estimated census metric);
h) comparing, by the second network, the estimated census metric with a corresponding metric from the census data to generate a difference there between (Brewer in Section IV.A. discloses that the model is trained using “L1 loss [i.e., mean absolute error, (MAE)]” where “models are trained until there is no improvement in validation accuracy after five epochs”. The L1 function computes the absolute difference between the predicted census metric and the actual census data metric).
Brewer does not explicitly teach all of a second network comprising a recurrent neural network inclusive of a fully connected layer trained with census data collected for a geographic region; c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion; g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold;
However, Mnih teaches a second network comprising a recurrent neural network inclusive of a fully connected layer trained with census data collected for a geographic region (Mnih in Abstract discloses “a novel recurrent neural network model” that processes sequential image regions);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Mnih in Abstract discloses a model that processes “selected regions at high resolution” where “the proposed model has a degree of translation invariance built-in, but the amount of computation it per forms can be controlled independently of the input image size”. The model processes selected image regions to generate feature outputs);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Mnih in Abstract discloses that the RNN model extracts “information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution”, where the sequential processing generates task-specific outputs);
i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold (Mnih in Abstract discloses a model that “adaptively select[s] a sequence of regions or locations” and “can be trained using reinforcement learning methods to learn task-specific policies”; the RNN adaptively selects a sequence of image regions, maintaining state across selections).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer by incorporating a recurrent visual-attention network that adaptively selects a sequence of image regions or locations and processes the selected regions at high resolution that is taught by Mnih, since both reference are analogous art in the field of neural-network-based computer vision and image analysis; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer’s satellite-image census-metric prediction with Mnih’s recurrent attention/location-selection model yields the predictable result of adaptively selecting and revising image regions for high-resolution feature processing, thereby reducing unnecessary computation and maintaining task-specific prediction performance.
Brewer in view of Mnih does not explicitly teach all of c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion; g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold.
However, Park teaches e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions (Park in Abstract discloses a deep learning model that predicts economic indicators by “aggregating traits observed from multiple levels of geographical units,” wherein “these images display detailed terrain over large areas and allowing zooming into smaller neighborhoods”; Park’s multi-level geographic aggregation teaches processing satellite imagery at different spatial resolutions (zoom levels) centered on the same geographic area for economic indicator prediction.);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Park in Abstract discloses a “deep learning model” that processes imagery at “multiple levels of geographical units” to extract features, where “the model first measures hyperlocal economy over small communities” and “the next step extracts district-level features by summarizing interconnection among hyperlocal economies”. Each level generates feature outputs indictive of features at that geographical resolution);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Park in ¶ discloses that “the model estimates economic indicators of districts via aggregating the hyperlocal and district information”, where “our new multi-level learning model substantially outperforms strong baselines in predicting key indicators such as population, purchasing power, and energy consumption”. Multi-level features are aggregated to generate estimated economic metrics).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer in view of Mnih by generating and processing satellite-image portions at multiple geographic levels/resolutions that is taught by Park, since both reference are analogous art in the field of geospatial/satellite-image analysis for estimating census indicators; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer in view of Mnih’s adaptive image-region processing with Park’s multi-level geospatial aggregation yields the predictable result of generating feature outputs from different geographic resolutions and aggregating those outputs, thereby improving census/economic metric estimation by capturing both local and broader-area context.
Brewer in view of Mnih, further in view of Park does not explicitly teach all of c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold.
However, Ayush teaches e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions (Ayush in Abstract discloses “a reinforcement learning approach in which free low resolution imagery is used to dynamically identify where to acquire costly high-resolution images”, where the RL agent selects the same spatial location for both low-resolution analysis and high-resolution acquisition);
i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold (Ayush in Abstract discloses “a reinforcement learning approach in which free low resolution imagery is used to dynamically identify where to acquire costly high-resolution images, prior to performing a deep learning task on the high-resolution images”; RL-based dynamic spatial location selection for poverty prediction, where the agent iteratively identifies optimal sampling locations based on prediction performance).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer in view of Mnih, further in view of Park by incorporating a reinforcement-learning approach to dynamically identify where to acquire or process high-resolution images that is taught by Ayush, since both reference are analogous art in the field of geospatial/satellite-image analysis for estimating census indicators; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer in view of Mnih, further in view of Park’s geospatial prediction framework with Ayush’s dynamic high-resolution location-selection approach yields the predictable result of moving/revising a selected centroid or sampling location to obtain more informative high-resolution image portions, thereby reducing acquisition/computation cost while maintaining or improving prediction performance.
Brewer in view of Mnih, further in view of Park, further in view of Ayush does not explicitly teach all of c) increasing, by the processor, the brightness value for each pixel in the cropped satellite image to generate a brightened satellite image.
However, the Examiner takes Official Notice. For “c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image”, the Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include image brightness enhancement. The Examiner takes Official Notice that image brightness enhancement was a well-known, standard preprocessing technique in computer vision. A PHOSITA would have recognized that satellite image data is commonly compressed in terms of dynamic range, and that increasing brightness is a predictable preprocessing step. The specific multiplication factor of at least 2 is a predicable parameter selection within the range of standard brightness enhancement techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 12
Regarding Claim 12, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 11, wherein the geographic region has a governing body associated therewith (Brewer in page 5670 discloses “Census data for the United States are acquired through the ACS. Every year, the U.S. Census Bureau contacts approximately 3.5 million households (1 in 40 total households) across the country to participate in the ACS”), and
wherein the census data is collected by the governing body (Brewer in page 5670 discloses “Census data for the United States are acquired through the ACS. Every year, the U.S. Census Bureau contacts approximately 3.5 million households (1 in 40 total households) across the country to participate in the ACS”).
Claim 13
Regarding Claim 13, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 11, wherein the selected region is selected from the group consisting of at least one of a state, a province, a city, a county, and a municipality (Brewer in page 5669 left column discloses “94 of the 100 largest U.S. cities by gross domestic product (GDP)”).
Claim 14
Regarding Claim 14, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 11, wherein the second portion is in a range of 50% to 80% smaller than the first portion (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include parameter selection for multi-resolution sampling. The Examiner takes Official Notice that multi-resolution sampling at different scales was a well-known, standard technique in computer vision, and that a 50-80% reduction in area is a predictable scale range for capturing coarser-resolution contextual features alongside finer-resolution detail. A PHOSITA would have recognized that this range represents a conventional trade-off between spatial coverage and feature resolution in multi-scale image analysis.).
Claim 15
Regarding Claim 15, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 11, wherein the second portion is approximately 75% smaller than the first portion (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include parameter selection for multi-resolution sampling. The Examiner takes Official Notice that multi-resolution sampling at different scales was a well-known, standard technique in computer vision, and that a 75% reduction in area is a predictable scale range for capturing coarser-resolution contextual features alongside finer-resolution detail. A PHOSITA would have recognized that this range represents a conventional trade-off between spatial coverage and feature resolution in multi-scale image analysis.).
Claim 16
Regarding Claim 16, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 11, wherein the centroid comprises a latitude and longitude in the selected region (Brewer in page 5671 discloses geographic boundaries from TIGER archive shapefiles, which define neighborhoods by latitude/longitude polygon coordinates; the centroid of each polygon is a latitude/longitude point).
Claim 17
Regarding Claim 17, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 11, wherein the step of moving includes applying a Gaussian distribution function to govern a distance between the location of the centroid and the revised location (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to apply a Gaussian distribution function to govern the distance between the current centroid location and the revised centroid location. The Examiner takes Official Notice that the Gaussian distribution was a well-known, standard mathematical mechanism for controlling step size in iterative optimization algorithms. A PHOSITA would have recognized that a Gaussian-distributed step size provides smooth, bounded movement that converges predictably, and that applying it to govern aspatial displacement in an iterative refinement process is a predictable engineering choice.).
Claim 18
Regarding Claim 18, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 11, wherein the first output and the second output comprise vector outputs (Brewer in page 5673 left column discloses that ResNet50 generates feature vectors, “2048-D representation” per patch in the semi-supervised approach, and feature space outputs in the supervised approach).
Claim 19
Regarding Claim 19, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches A method, comprising:
by an interactive neural network inclusive of a first network trained with ImageNet (Brewer in page 5672 left column discloses “The ResNet50-based architecture of Fig. 5 has its base model pre-trained on ImageNet.”),
a) obtaining a satellite image of the geographic region (Brewer in Fig. 2 to Fig.4 and page 5670 right column discloses obtaining aerial imagery of geographic regions);
b) cropping, by a processor coupled to the interactive neural network, the satellite image to define a cropped satellite image inclusive of a selected region of the geographic region (Brewer in page 5671 discloses cropping/patching satellite imagery to define image patches corresponding to selected geographic regions using TIGER/line shapefiles, wherein each pixel inherently possesses brightness values),
wherein the selected region comprises at least one of a state, a province, a city, a county, and a municipality of the geographic region (Brewer in page 5669 left column discloses “94 of the 100 largest U.S. cities by gross domestic product (GDP)”), and
wherein each pixel of the cropped satellite image has a brightness value (Brewer in page 5671 discloses cropping/patching satellite imagery to define image patches corresponding to selected geographic regions using TIGER/line shapefiles, wherein each pixel inherently possesses brightness values);
d) sampling, by the processor, a first portion of the brightened satellite image wherein the first portion has a centroid defined by a location in the selected region (Brewer in Fig 3. and Fig. 4 discloses sampling image portions (patches) from satellite imagery corresponding to geographic regions, with each patch centered on a centroid of a Census block group);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Brewer in Abstract and page 5672 left column discloses a “ResNet50-based architecture” is used to extract features from image portions, where “features extracted from the imagery can accurately estimate the density” of neighborhoods. The ResNet50 processes image patches to generate feature outputs indicative of features in the imagery);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Brewer in Section IV.A. discloses that “seven fully connected layers are added after the base model to gradually scale down the feature space to a single output estimation”, where the output estimation is one of “density, MHI, education” at the U.S. Census block level. The fully connected layers process the CNN feature outputs to generate an estimated census metric);
h) comparing, by the second network, the estimated census metric with a corresponding metric from the census data to generate a difference there between (Brewer in Section IV.A. discloses that the model is trained using “L1 loss [i.e., mean absolute error, (MAE)]” where “models are trained until there is no improvement in validation accuracy after five epochs”. The L1 function computes the absolute difference between the predicted census metric and the actual census data metric).
Brewer does not explicitly teach all of a second network comprising a recurrent neural network inclusive of a fully connected layer trained with census data collected for a geographic region; c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion; g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold;
However, Mnih teaches a second network comprising a recurrent neural network inclusive of a fully connected layer trained with census data collected for a geographic region (Mnih in Abstract discloses “a novel recurrent neural network model” that processes sequential image regions);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Mnih in Abstract discloses a model that processes “selected regions at high resolution” where “the proposed model has a degree of translation invariance built-in, but the amount of computation it per forms can be controlled independently of the input image size”. The model processes selected image regions to generate feature outputs);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Mnih in Abstract discloses that the RNN model extracts “information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution”, where the sequential processing generates task-specific outputs);
i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold (Mnih in Abstract discloses a model that “adaptively select[s] a sequence of regions or locations” and “can be trained using reinforcement learning methods to learn task-specific policies”; the RNN adaptively selects a sequence of image regions, maintaining state across selections).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer by incorporating a recurrent visual-attention network that adaptively selects a sequence of image regions or locations and processes the selected regions at high resolution that is taught by Mnih, since both reference are analogous art in the field of neural-network-based computer vision and image analysis; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer’s satellite-image census-metric prediction with Mnih’s recurrent attention/location-selection model yields the predictable result of adaptively selecting and revising image regions for high-resolution feature processing, thereby reducing unnecessary computation and maintaining task-specific prediction performance.
Brewer in view of Mnih does not explicitly teach all of c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion; g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold.
However, Park teaches e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions (Park in Abstract discloses a deep learning model that predicts economic indicators by “aggregating traits observed from multiple levels of geographical units,” wherein “these images display detailed terrain over large areas and allowing zooming into smaller neighborhoods”; Park’s multi-level geographic aggregation teaches processing satellite imagery at different spatial resolutions (zoom levels) centered on the same geographic area for economic indicator prediction.);
f) processing, by the first network, the first portion to generate a first output indicative of features in the first portion and the second portion to generate a second output indicative of features in the second portion (Park in Abstract discloses a “deep learning model” that processes imagery at “multiple levels of geographical units” to extract features, where “the model first measures hyperlocal economy over small communities” and “the next step extracts district-level features by summarizing interconnection among hyperlocal economies”. Each level generates feature outputs indictive of features at that geographical resolution);
g) processing, by the second network and the third network, the first output and the second output to generate an estimated census metric associated with the centroid (Park in ¶ discloses that “the model estimates economic indicators of districts via aggregating the hyperlocal and district information”, where “our new multi-level learning model substantially outperforms strong baselines in predicting key indicators such as population, purchasing power, and energy consumption”. Multi-level features are aggregated to generate estimated economic metrics).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer in view of Mnih by generating and processing satellite-image portions at multiple geographic levels/resolutions that is taught by Park, since both reference are analogous art in the field of geospatial/satellite-image analysis for estimating census indicators; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer in view of Mnih’s adaptive image-region processing with Park’s multi-level geospatial aggregation yields the predictable result of generating feature outputs from different geographic resolutions and aggregating those outputs, thereby improving census/economic metric estimation by capturing both local and broader-area context.
Brewer in view of Mnih, further in view of Park does not explicitly teach all of c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image; e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions; i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold.
However, Ayush teaches e) generating, by the processor, a second portion of the brightened satellite image centered on the centroid wherein the first portion and the second portion have different resolutions (Ayush in Abstract discloses “a reinforcement learning approach in which free low resolution imagery is used to dynamically identify where to acquire costly high-resolution images”, where the RL agent selects the same spatial location for both low-resolution analysis and high-resolution acquisition);
i) moving, by the second network, the location of the centroid to a revised location in the selected region and repeating steps d) through h) until the difference is less than a prescribed threshold (Ayush in Abstract discloses “a reinforcement learning approach in which free low resolution imagery is used to dynamically identify where to acquire costly high-resolution images, prior to performing a deep learning task on the high-resolution images”; RL-based dynamic spatial location selection for poverty prediction, where the agent iteratively identifies optimal sampling locations based on prediction performance).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Brewer in view of Mnih, further in view of Park by incorporating a reinforcement-learning approach to dynamically identify where to acquire or process high-resolution images that is taught by Ayush, since both reference are analogous art in the field of geospatial/satellite-image analysis for estimating census indicators; thus, one of ordinary skilled in the art would be motivated to combine the references since Brewer in view of Mnih, further in view of Park’s geospatial prediction framework with Ayush’s dynamic high-resolution location-selection approach yields the predictable result of moving/revising a selected centroid or sampling location to obtain more informative high-resolution image portions, thereby reducing acquisition/computation cost while maintaining or improving prediction performance.
Brewer in view of Mnih, further in view of Park, further in view of Ayush does not explicitly teach all of c) increasing, by the processor, the brightness value for each pixel in the cropped satellite image to generate a brightened satellite image.
However, the Examiner takes Official Notice. For “c) multiplying, by the processor, the brightness value for each pixel in the cropped satellite image by a factor of at least 2 to generate a brightened satellite image”, the Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include image brightness enhancement. The Examiner takes Official Notice that image brightness enhancement was a well-known, standard preprocessing technique in computer vision. A PHOSITA would have recognized that satellite image data is commonly compressed in terms of dynamic range, and that increasing brightness is a predictable preprocessing step. The specific multiplication factor of at least 2 is a predicable parameter selection within the range of standard brightness enhancement techniques.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 20
Regarding Claim 20, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 19, wherein the second portion is in a range of 50% to 80% smaller than the first portion (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include parameter selection for multi-resolution sampling. The Examiner takes Official Notice that multi-resolution sampling at different scales was a well-known, standard technique in computer vision, and that a 50-80% reduction in area is a predictable scale range for capturing coarser-resolution contextual features alongside finer-resolution detail. A PHOSITA would have recognized that this range represents a conventional trade-off between spatial coverage and feature resolution in multi-scale image analysis.).
Claim 21
Regarding Claim 21, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 19, wherein the second portion is approximately 75% smaller than the first portion (The Examiner finds that, at the time of filing, it would have been obvious to a person having ordinary skill in the art to include parameter selection for multi-resolution sampling. The Examiner takes Official Notice that multi-resolution sampling at different scales was a well-known, standard technique in computer vision, and that a 75% reduction in area is a predictable scale range for capturing coarser-resolution contextual features alongside finer-resolution detail. A PHOSITA would have recognized that this range represents a conventional trade-off between spatial coverage and feature resolution in multi-scale image analysis.).
Claim 22
Regarding Claim 22, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 19, wherein the centroid comprises a latitude and longitude in the selected region (Brewer in page 5671 discloses geographic boundaries from TIGER archive shapefiles, which define neighborhoods by latitude/longitude polygon coordinates; the centroid of each polygon is a latitude/longitude point).
Claim 23
Regarding Claim 23, Brewer in view of Mnih, further in view of Park, further in view of Ayush teaches The method of claim 19, wherein the first output and the second output comprise vector outputs (Brewer in page 5673 left column discloses that ResNet50 generates feature vectors, “2048-D representation” per patch in the semi-supervised approach, and feature space outputs in the supervised approach).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUSTIN P CASCAIS whose telephone number is (703) 756-5576. The examiner can normally be reached Monday-Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mr. O'Neal Mistry can be reached on (313) 446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.P.C./Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Date: 6/22/2026
Prosecution Insights