Detailed Action
1. Claims 1-14 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless -
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
3. Claims1-3, 5 and 9-14 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mark J. Carlotto ( hereafter Carlotto), “Detection and Analysis of Change in Remotely Sensed Imagery with Application to Wide Area Surveillance” IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 1, pub. JANUARY 1997
As to claim 1, Carlotto teaches An aerial image change detection apparatus ( Abstract, Detection and Analysis of Change in Remotely Sensed Imagery with Application to Wide Area Surveillance), comprising:
an image acquiring unit which acquires N-number of first aerial images and M- number of second aerial images, wherein the first aerial images are aerial images of a same area in a first period, the second aerial images are aerial images of the same area as the first aerial images in a second period subsequent to the first period, and N and M are both integers larger than or equal to 1 (Figs.1 and 2, page 193 left col., 2nd par., A new approach to wide area surveillance (WAS) is described that is based on the detection and analysis of changes across two or more images over time. The Authors approaches to WAS involved the analysis of changes between all pairs of input images. For N input images ,
x
n
i
,
j
,
0
≤
n
≤
N
,
there are
N
(
N
-
1
)
2
total difference images . Consider an example (Fig. 1) where three images acquired at times t1,t2, and t3 and have captured two events (Fig. 1(a)). The and N and M set of of images corresponds to the N input images and N>0);
a change candidate extracting unit which obtains, with respect to a plurality of image pairs among N x M-number of image pairs of the first aerial images and the second aerial images, a change index for each pixel that represents a change in a pixel value between one first aerial image and one second aerial image and (Figs.1 and 2, page 193 section C Temporal Segmentation; For N input images ,
x
n
i
,
j
,
0
≤
n
≤
N
,
there are
N
N
-
1
2
total difference images, where the total difference images is given by
n
m
i
,
j
,
0
≤
m
≤
N
,
0
≤
n
≤
N
, where the total difference images determined based on patterns of change across images over time. The patterns of change are represented by an image of label vectors
⋀
i
,
j
=
λ
n
i
,
j
.At a particular pixel location, two elements of the label vector are equal if there is no change between the corresponding times at that pixel
λ
n
i
,
j
=
λ
m
i
,
j
otherwise, the two elements are different
λ
n
i
,
j
≠
[Symbol font/0x20]
λ
m
i
,
j
.
The change index corresponds to the total difference images
n
m
i
,
j
g
i
v
e
n
b
y
e
q
u
a
t
i
o
n
6
) which adopts a pixel of which the change index is larger than a change threshold as a change location candidate (Equations 6 and 9, page 193 left col., 2nd par., The total difference image
∈
=
∈
F
i
,
j
-
∈
B
i
,
j
, and the thresholds
τ
m
n
are computed from the histograms of the total difference images given by equation (6), where
∈
>
τ
m
n
)
wherein the change threshold is chosen in accordance with the pair of the first aerial image and the second aerial image (Equation 9, page 193 right col., 1st and 2nd pars., although this segmentation algorithm can provide insight into the kinds of changes occurring in different parts of the image, its performance depends on the thresholds that, in turn, depend on the value the histogram given by equation 9); and
a change location extracting unit which determines, as a change location, a pixel considered to be a change location candidate in a prescribed percentage or more of the plurality of image pairs ( Equation 9, page 193 right col., 1st and 2nd pars ., the thresholds are computed from the histograms of the total difference images. Each is chosen to satisfy
p
0
=
∑
∈
>
τ
m
i
n
p
m
n
(
∈
)
, where
p
m
n
(
∈
)
is the histogram of the total difference image
∈
m
n
i
,
j
, and
p
0
is the fraction of the image that is assumed Patterns of change. Thus, the histogram of the total image p0 given by equation 9 and equation 6 measures the change locations candidates ( pixels) )
As to claim 2, Carlotto teaches the change index of a pixel is a value in accordance with a ratio of a change in a pixel value of the pixel to a mean of changes in pixel values in a peripheral region of the pixel(Equation 9 page 193 right col. 2nd par., the total difference image
∈
=
∈
F
i
,
j
-
∈
B
i
,
j
, and the thresholds
τ
m
n
are computed from the histograms of the total difference images given by equation (6), where
∈
>
τ
m
n
. Thistograms of the total difference images inherently teaches the mean of the difference image ( mean of change). Specifically from the histogram the mean value of estimated simply by calculating the midpoint of each bin of the histogram, multiplying it by the bin's frequency, summing these totals, and dividing by the total number of data points)
As to claim 3, Carlotto teaches the change candidate extracting unit obtains the change index after adjusting at least one of the first aerial image and the second aerial image so that a mean value and a variance of pixel values of the first aerial image and the second aerial image are consistent (page 191, left col. Adaptive techniques use information over larger areas (e.g., within a sliding window) to model and predict two or more images from one another. The difference between the actual and predicted image is used as a measure of change. For two images (random variables) (x1, x2)and times (t1, t2) on and ,where acquired at t2>t1, we seek an estimate x2 based on x1 that minimize the Mean Square Error (MSE) given by equation 1)
As to claim 5, Carlotto teaches the change candidate extracting unit adopts a pixel as a change location candidate regardless of a value of the change index when a change of a pixel value of the pixel is larger than a prescribed value (Equations 6 and 9, page 193 left col., 2nd par The patterns of change are represented by an image of label vectors
⋀
i
,
j
=
λ
n
i
,
j
.At a particular pixel location, two elements of the label vector are equal if there is no change between the corresponding times at that pixel
λ
n
i
,
j
=
λ
m
i
,
j
otherwise, the two elements are different
λ
n
i
,
j
≠
[Symbol font/0x20]
λ
m
i
,
j
.
The change index corresponds to the total difference images
n
m
i
,
j
g
i
v
e
n
b
y
e
q
u
a
t
i
o
n
6
. The total difference image
∈
=
∈
F
i
,
j
-
∈
B
i
,
j
, and the thresholds
τ
m
n
are computed from the histograms of the total difference images given by equation (6), where
∈
>
τ
m
n
.
Although this segmentation algorithm can provide insight into the kinds of changes occurring in different parts of the image, its performance depends on the thresholds that, in turn, depend on the value the histogram given by equation 9).
As to claim 9, Carlotto teaches wherein the image acquiring unit: acquires a partial area of a first large-area aerial image in the first period as the first aerial image; and acquires a partial area of a second large-area aerial image in the second period as the second aerial image (page 191, left col. Adaptive techniques use information over larger areas (e.g., within a sliding window) to model and predict two or more images from one another).
As to claim 10, Carlotto teaches the image acquiring unit: acquires the N-number of the first aerial images from a top N-number of the first large-area aerial images with a small cloud coverage in the partial area among more than N- number of first large-area aerial images in the first period; and acquires the M-number of the second aerial images from a top M-number of the second large-area aerial images with a small cloud coverage in the partial area among more than M-number of second large-area aerial images in the second period (Page 197- right col.- Page 198 The two images were taken about a month apart in 3/30/88 and 5/1/88. Except for the clouds, there are relatively few changes on the ground. The clouds in (Fig. 7(b)) significantly increase the scatter in the joint histogram (Fig. 7(c)). They effectively pull the linear model away from the hypothetical regression line, assuming there were no clouds (dotted line) to the actual regression line with clouds (solid line). The absolute value of the total difference image is shown in (Fig. 7(d)) and is dominated by changes dues to the clouds. Fig. 8 shows how the nonlinear prediction algorithm can better adapt to large changes due to cloud.
As to claim 11, Carlotto teaches a change location is detected for each of a plurality of divided areas obtained by dividing an area of interest, and a change in the area of interest between the first period and the second period is obtained ( Page 200 section C, Temporal segmentation was performed on seven of the 10 over the same 400X400 pixel region shown in Figs. 6±8. First, total difference images were computed using the nonlinear prediction algorithm. The labeling algorithm (10) was then applied to the change images of 400X400 pixel region).
As to claim 12, Carlotto teaches an output unit which outputs, in superposition, at least any of the N-number of first aerial images and the M-number of second aerial images and the change locations ( page 200 right col., last par., - page 201 left col. 1st par., Fig. 12 shows the full scene outputs from the Delta filter for two images (Fig. 12(a)) and eight images (Fig. 12(b)) Fig. 12(a) is the total difference image between the 6/24/87 and 3/30/88 images and represents a two image change detection result. Bright areas indicate an increase in the brightness. In dark areas, the total difference has decreased and includes areas where the water level has risen to cover the land and vegetation has appeared or developed more fully. In Fig. 12(b), there are relatively few bright areas where the Delta filter responds strongly other than over the facility itself. The output from the Delta filter after it has been thresholder and smoothed is shown in Fig. 13(a). The largest detection is highlighted and overlaid on the 7/10/90 image in Fig. 13(b).
Claim 13 is rejected the same as claim 1 except claim 13 is directed to a method claim. All the limitations of claim 13 are addressed in claim 1. Thus, argument analogous to that presented above for claim 1 is applicable to claim 13.
As to claim 14, Carlotto teaches A computer-readable medium non-transitorily storing a program for causing a computer to execute the steps (Figs. 3-12, page 193 right col., 2nd par., page 201 right col. 3rd par., Carlotto specifically teaches various computer excitable geocoding ( i.e. computer program) that includes detection algorithm, nonlinear prediction algorithm, image labeling algorithm, linear and nonlinear change detection algorithm, and the experimental result ( computer simulation results) obtained by executing the various geocoding (see Figs. 3-12);
regarding the remaining limitation of claim 14, all the remaining limitations of claim are addressed in claim 1. Thus, argument analogous to that presented above for claim 1 is applicable to the remaining limitation of claim 14.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. Claim 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Carlotto
“Detection and Analysis of Change in Remotely Sensed Imagery with Application to Wide Area Surveillance” in view of QUI et al.,(hereafter QUI), CN 107154026 A, pub. 09/12/2017
Regarding claim 6, while Carlotto teaches the limitation of claim 1 but fails to teach the limitation claim 6 .
On the other hand in the same field of image processing, specifically to a shadow eliminating road-based adaptive brightness elevation model of the method of QUI teaches the change candidate extracting unit extracts a pixel obtained by performing area opening processing with respect to a change location candidate as a final change location candidate in the pair of the first aerial image and the second aerial image ( page 3rd par., using the Otsu method, namely the maximum inter-class variance method to calculate and divide the threshold value of the SDI enhanced image; obtaining the binary mask image, the pixel point less than the threshold value is black, representing the shadow area, the part greater than the threshold value is white, representing the non-shadow area; at last, using mathematical morphology opening operation to fill the hole, using closed operation to remove micro burr and debris, after morphological optimization processing, obtaining shadow detection final result)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a well- known morphology opening
operation to fill the hole and closed operation to remove noise taught by QUI into Carlotto.
The suggestion/motivation for doing so would have been to allow user of Carlotto to
obtain a clear shadow image by removing noise from the shadow and restoring the shape of the shadow image.
As to claim 7, QUI teaches the change candidate extracting unit obtains a shadow area in the pair of the first aerial image and the second aerial image and the shadow area is excluded from the change location candidates (page 3 1st par., shadow land deletion. by analyzing the spectrum of the remote sensing image of the unmanned aerial vehicle, detail and so on, characteristics a method for recovering scene color and texture in shadow area of remote sensing image of unmanned aerial vehicle, so that the shadow area scene and the real color of the ground object have consistency.).
5. Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Carlotto
“Detection and Analysis of Change in Remotely Sensed Imagery with Application to Wide Area Surveillance” in view of ZHONG et al., (hereafter ZHONG), CN 114648709 A , pub. 06-21-2022
Regarding claim 4, while Carlotto teaches the limitation of claim 1 but fails to teach the limitation claim 4 .
On the other hand in the same field of endeavor a methods determining image difference information of ZHONG teaches the change threshold is set for each pair as a greater value among a prescribed value and a value corresponding to a higher first prescribed percentage of change indices calculated from the first aerial image and the second aerial image(Claim 8, the pair of each sub-image for similar matching, based on the matching of the sub-image pair determining the matching image, further comprising: if the average value of the maximum pixel difference of the three pixel difference average value of a certain sub-image pair is greater than the preset pixel difference threshold value, determining the unmanned aerial vehicle characteristic vector of the sub-image to the unmanned aerial vehicle sub-image and the template characteristic vector of the corresponding template sub-image; if the distance between the unmanned aerial vehicle feature vector and the template feature vector is less than or equal to the preset distance threshold value, then determining the sub-image pair is matched sub-image pair)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a method of images similarity by comparing on the distance between images feature vectors taught by ZHONG into Carlotto.
The suggestion/motivation for doing so would have been feature-based methods are more robust, efficient, and better at identifying content similarity despite distortions.
6. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Carlotto
“Detection and Analysis of Change in Remotely Sensed Imagery with Application to Wide Area Surveillance” in view of QUI, CN 107154026 A, farther in view of ANDERSON et al., ( hereafter ANDERSON), WO 2021243360 A1, pub. 12/02/2021
Regarding claim 8, while the combination of Carlotto and QUI teaches the limitation of claim 7 but fails to teach the limitation claim 8 .
On the other hand in the same field of endeavor a methods to perform an automated visual-inspection of components using image processing of ANDERSON teaches the change candidate extracting unit obtains a dark area of which a pixel value in both the first aerial image and the second aerial image is smaller than or equal to a first threshold and obtains a sum area of a first area created by expanding the dark area in the first aerial image to a range in which a pixel value is smaller than or equal to a second threshold that is larger than the first threshold and a second area created by expanding the dark area in the second aerial image to a range in which a pixel value is smaller than or equal to the second threshold ( Fig.6B, [0105] , If a determination is made at operation 607C that the defect is “large,” then erosion and/or dilation steps are performed at operation 607D. The erosion and dilation steps are performed potentially to link pixels identified as a portion of the large defect together. For example, an erosion step increases an area of black regions from the threshold
image (from operation 607A) to link the black regions together to group the black regions as a larger defect. Conversely, the dilation step reduces an area of black regions from the threshold image that are determined not to belong to a larger defect.
Consequently, in this embodiment, dilation increases the size of white areas and erosion increases size of black area)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a method of increasing an area of black regions from the threshold image to link a plurality of black regions together to group the black regions as a large black region using a well- known dilation operator taught by ANDERSON into modified Carlotto.
The suggestion/motivation for doing so would have been to allow user of modified Carlotto to connect nearby black regions that are separated by small white gaps, and effectively grouping them into a single black region by replacing small white spots inside black regions. Thus, the dilation operation making objects more solid (uniform color) and easier to detect.
Prior art of record but not applied in the rejection
“Aerial image change detection using dual regions of interest networks”, Neuro computing Volume 349, 15 July 2019, Pages 190-201, to Pengcheng Han et al., disclosed:
An end-to-end novel deep learning model for semantic change detection issue on aerial images. The proposed dual regions of interest networks (DRoINs) mainly contain three functional blocks: feature extracted networks, change proposal networks, and difference judgement networks. The DRoINs is designed to share the parameters of model between input paired images, which could reduce the distance within classes and increase that between classes, and improve the feature representative to make change discrimination more accurate. This technique is focused on the object-based change detection problem. In practical application, it could deal with some challenges. Firstly, it could cope with the conditions that paired images are not strictly aligned pixel-to-pixel, after all, we could not make sure that cameras are always mounted at stationary objects, for example to obtain the aerial images with cameras mounted on UAVs. Secondly, season changes, light change, and noise disturbance could be handled, in which conditions the proposed technique is more robust. Thirdly, the DRoINs output whether the change happens in regions of interest while giving the categories of paired images. The last but not the least, to research our defined change detection problem, a novel pipeline is proposed to create the aerial dataset for our defined change detection problem.
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday -Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.217-919 (tool-free)
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699
Prosecution Insights