DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application is a National Stage application of PCT/CN2023/081019 filed 03/13/2023. Priority to CHINA 202210239630.9 with a priority date of 03/11/2022 is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Information Disclosure Statement
The IDS dated 12/10/2024 and 03/05/2026 have been considered and placed in the application file.
Claim Objections
Claims 4 -11 are objected to because of the following informalities:
Claim 4 is objected to because "taking the each of the scale branches" in Line 7 is unclear. The examiner suggests that the claim be changed to reach "taking each of the scale branches" Claims 5-11 depend on Claim 4, therefore they are also objected to. Appropriate correction is required.
Claim 16 is objected to because of the following informalities:
Claim 16 is objected to because "specified dimensions" are not clearly defined in the claims or specification. Based on the claim language and specification it is unclear how much the specified dimensions would affect the training images and therefore the result would not be able to be replicated by one of ordinary skill in the art. Appropriate correction is required.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation using the
plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
Under MPEP 2143.03, "All words in a claim must be considered in judging the patentability of that claim against the prior art." In re Wilson, 424 F.2d 1382, 1385, 165 USPQ 494, 496 (CCPA 1970). As a general matter, the grammar and ordinary meaning of terms as understood by one having ordinary skill in the art used in a claim will dictate whether, and to what extent, the language limits the claim scope. Language that suggests or makes a feature or step optional but does not require that feature or step does not limit the scope of a claim under the broadest reasonable claim interpretation. In addition, when a claim requires selection of an element from a list of alternatives, the prior art teaches the element if one of the alternatives is taught by the prior art. See, e.g., Fresenius USA, Inc. v. Baxter Int’l, Inc., 582 F.3d 1288, 1298, 92 USPQ2d 1163, 1171 (Fed. Cir. 2009).
Claim 16 recite "more than one of" then listing “clarity, color, contrast and noise”. Since “more than one of” is disjunctive, any two of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only two elements are required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 12-16, 18, 19, and 21 are rejected under 35 U.S.C. 103 as unpatentable over Chen et al (CN113298740A using machine translation from espace.net and google translate for figures, hereafter referred to as Chen) in view of Ma et al (Ma, Jinming, et al. "MAFF-Net: Multi-attention guided feature fusion network for change detection in remote sensing images." Sensors 22.3 (2022): 888, hereafter referred to as Ma).
Regarding Claim 1, Chen teaches an image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method), comprising:
obtaining an initial image to be processed (Chen Pg 2 ¶02 discloses the initial image to be processed is and LDR image);
inputting the initial image into an image enhancement model (Chen Pg 1 ¶09 discloses inputting the image to be processed into a trained image enhancement model) obtained by pre-training (Chen Pg 5 ¶10 discloses the image enhancement model can be pre-trained by the image processing device), wherein the image enhancement model (Chen Pg 1 ¶09, Pg 2 ¶03, Pg 2 ¶07 discloses a trained image enhancement model) comprises a multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features);
performing a multi-scale feature extraction on an input image through the multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features)
wherein the input image is obtained based on the initial image (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features or the initial image); and
obtaining an image of which an image quality is enhanced (Chen Pg 1 ¶08-¶09 and Pg 2 ¶01 discloses the output being an enhanced image) and the initial image (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features).
Chen does not explicitly disclose to obtain initial feature maps of multiple scales performing a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps, and performing a fusion based on the multiple intermediate feature maps to obtain an output feature map of the multi-scale feature fusion network, and based on the output feature map of the multi-scale feature fusion network.
Ma is in the same field of using feature fusion networks for image enhancement. Further, Ma teaches to obtain initial feature maps of multiple scales (Ma, Abstract and Section 2.3 ¶03 discloses feature maps at different scales of one image), performing a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps), and performing a fusion based on the multiple intermediate feature maps to obtain an output feature map of the multi-scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output),
and based on the output feature map of the multi-scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen by incorporating the implementation of multiple scales into the network architecture to obtain multiple feature maps to create the multi scale feature fusion network as taught by Ma; to make an invention that can use the network to further refine the feature resolution of the images; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to address the key issues of remote sensing images by extracting the rich feature information of HR remote sensing images, better focusing on the change regions, avoiding the interference of other factors, and reducing the interference of pseudo-changes (Ma, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Chen in view of Ma teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 1, wherein the performing a multi-scale feature extraction on an input image through the multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features) to obtain initial feature maps of multiple scales (Ma, Abstract and Section 2.3 ¶03 discloses feature maps at different scales of one image) comprises:
performing down-samplings on the input image according to a plurality of preset multiples (Chen Pg 1 ¶10 and Pg 4 ¶01 discloses M+1 feature extraction modules include different numbers of Downsampling operation, Pg 4 ¶02 discloses down sampling the initial image into three scales, the large scale, the intermediate scale and the small scale) respectively to obtain the initial feature maps of the multiple scales (Chen Pg 4 ¶02 discloses down sampling the initial image into three scales, the large scale, the intermediate scale and the small scale to obtain the spatial features), wherein the preset multiples are lower than a preset threshold (Chen Pg 4 ¶02 discloses down sampling the initial image into three scales, the large scale, the intermediate scale and the small scale to obtain the spatial features, the large scale is the original image size as disclosed by Chen, therefore the examiner is interpreting the original image size to be the threshold and all preset multiples are lower than the original as disclosed in Chen Pg 4 ¶02). See Claim 1 for rationale, its parent claim.
Regarding Claim 12, Chen in view of Ma teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 1, wherein the performing a fusion based on the multiple intermediate feature maps to obtain an output feature map of the multi-scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output) comprises:
fusing the multiple intermediate feature maps (Ma Fig 1 discloses the fusion of the intermediate feature maps in the green boxes) to obtain a fusion feature map (Ma Pg 19 ¶01 discloses obtaining fusion feature maps), wherein a scale of the fusion feature map is same as that of the input image of the multi-scale feature fusion network (Ma Fig 2 and Pg 6 ¶03 and Pg 7 ¶02, discloses resizing the output to be the same size as that as the input); and
performing an element-wise sum fusion based on the fusion feature map and the input feature map of the multi-scale feature fusion network (Ma Fig 4 and Section 2.4 disclose four feature maps are subjected to element-wise summation and the final output feature map is obtained) and to obtain the output feature map of the multi- scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output). See Claim 1 for rationale, its parent claim.
Regarding Claim 13, Chen in view of Ma teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 12, wherein a way of performing the fusion based on the initial feature maps (Ma Fig 4 and Section 2.4 discloses that element wise summation is used on F1 the initial feature block) is same as a way of performing the fusion based on the multiple intermediate feature maps (Ma Fig 4 and Section 2.4 discloses that element wise summation is used on F1 the initial feature block and is also used on F2 and F4 the intermediate feature maps). See Claim 1 for rationale, its parent claim.
Regarding Claim 14, Chen in view of Ma teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 1,wherein a number of the multi-scale feature fusion network is multiple (Ma Fig 1 discloses the use of multiple Feature fusion model represented by the blue boxes), and the multiple multi-scale feature fusion networks are sequentially connected in series (Ma Fig 1 discloses that the four FFM models are connected in series to the last FFM models), wherein the input image of a first multi-scale feature fusion network is obtained based on the initial image (Ma Fig 1 discloses that T1 and T2 are the input images and are fed into the first FFM models), and the input image of a non-first multi-scale feature fusion network is obtained based on the output feature map of a previous multi-scale feature fusion network (Ma Fig 1 discloses the output of the four FFM being the input for the final FFM that produces the output). See Claim 1 for rationale, its parent claim.
Regarding Claim 15, Chen in view of Ma teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 1,wherein the image enhancement model is trained (Chen Pg 1 ¶09 discloses inputting the image to be processed into a trained image enhancement model) according to the following method:
obtaining training sample pairs (Chen Pg 6 ¶02 discloses a plurality of sample pairs for training), wherein each of the training sample pairs comprises an image quality enhanced sample and an image quality degraded sample with consistent image content (Chen Pg 6 ¶02 discloses each image sample pair includes an LDR image sample (representing the degraded sample) and an HDR image sample (representing an quality enhanced sample) corresponding to the LDR image sample), and a number of the training sample pairs is multiple (Chen Pg 6 ¶02 discloses a plurality of sample pairs for training); and
training a neural network model pre-built based on the training sample pairs and a preset loss function (Chen Pg 6 ¶03-¶04 discloses training the network based on the pixel values of the image pairs and Tanh_L1 loss function), and taking the trained neural network model as the image enhancement model (Chen Pg 6 ¶08 discloses Based on the training set and Tanh_L1, the gradient descent method can be used to iteratively train the initial image enhancement model. When the model converges (that is, the value of Tanh_L1 is not decreasing), the trained image enhancement model can be obtained). See Claim 1 for rationale, its parent claim.
Regarding Claim 16, Chen in view of Ma teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 15, wherein the obtaining training sample pairs (Chen Pg 6 ¶02 discloses a plurality of sample pairs for training) comprises:
obtaining image samples (Ma Fig 5 and Fig 7 discloses the image samples);
performing a degradation processing on each of the image samples (Chen Pg 6 ¶02 discloses each image sample pair includes an LDR image sample (representing the degraded sample)) according to specified dimensions to obtain the image quality degraded sample (Chen Pg 5 ¶09 discloses training in a set of low resolution image samples including blurred image samples), wherein the specified dimensions comprise more than one of clarity, color, contrast and noise (Chen Pg 5 ¶07 discloses the image enhancement or degradation to be defogging, noise removal, rain removal, super-resolution, decompression artifacts); and
taking the each of the image samples as the image quality enhanced sample (Chen Pg 6 ¶02 discloses each image sample pair an HDR image sample (representing an quality enhanced sample) corresponding to the LDR image sample), or, performing an enhancement processing on the each of the image samples (Chen Pg 6 ¶02 discloses each image sample pair includes a HDR image sample (representing an quality enhanced sample) corresponding to the LDR image sample) according to the specified dimensions to obtain the image quality enhanced sample (Chen Pg 5 ¶07 discloses the image enhancement or degradation to be defogging, noise removal, rain removal, super-resolution, decompression artifacts). See Claim 1 for rationale, its parent claim.
Regarding Claim 18, Chen teaches an electronic device (Chen Pg 2 ¶04-¶07 discloses an image enhancement device that provides a terminal device including a memory and processor where the memory is used to store a computer program that is then executed by the processor Pg 9 ¶03 discloses electronic hardware), wherein the electronic device comprises:
a processor (Chen Pg 2 ¶04-¶07 discloses an image enhancement device that provides a terminal device including a memory and processor where the memory is used to store a computer program that is then executed by the processor), and
a memory for storing executable instructions of the processor (Chen Pg 2 ¶04-¶07 discloses an image enhancement device that provides a terminal device including a memory and processor where the memory is used to store a computer program that is then executed by the processor),
wherein the processor is configured to read the executable instructions from the memory, and execute the executable instructions (Chen Pg 2 ¶04-¶07 discloses an image enhancement device that provides a terminal device including a memory and processor where the memory is used to store a computer program that is then executed by the processor) to:
obtain an initial image to be processed (Chen Pg 2 ¶02 discloses the initial image to be processed is and LDR image);
input the initial image into an image enhancement model (Chen Pg 1 ¶09 discloses inputting the image to be processed into a trained image enhancement model) obtained by pre-training (Chen Pg 5 ¶10 discloses the image enhancement model can be pre-trained by the image processing device), wherein the image enhancement model (Chen Pg 1 ¶09, Pg 2 ¶03, Pg 2 ¶07 discloses a trained image enhancement model) comprises a multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features);
perform a multi-scale feature extraction on an input image through the multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features) wherein the input image is obtained based on the initial image (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features or the initial image); and
obtain an image of which an image quality is enhanced (Chen Pg 1 ¶08-¶09 and Pg 2 ¶01 discloses the output being an enhanced image) and the initial image (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features).
Chen does not explicitly disclose to obtain initial feature maps of multiple scales, perform a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps, and perform a fusion based on the multiple intermediate feature maps to obtain an output feature map of the multi-scale feature fusion network and based on the output feature map of the multi-scale feature fusion network.
Ma is in the same field of using feature fusion networks for image enhancement. Further, Ma teaches to obtain initial feature maps of multiple scales (Ma, Abstract and Section 2.3 ¶03 discloses feature maps at different scales of one image), perform a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps), and perform a fusion based on the multiple intermediate feature maps to obtain an output feature map of the multi-scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output)
and based on the output feature map of the multi-scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen by incorporating the implementation of multiple scales into the network architecture to obtain multiple feature maps to create the multi scale feature fusion network as taught by Ma; to make an invention that can use the network to further refine the feature resolution of the images; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to address the key issues of remote sensing images by extracting the rich feature information of HR remote sensing images, better focusing on the change regions, avoiding the interference of other factors, and reducing the interference of pseudo-changes (Ma, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 19, Chen teaches a non-transitory computer-readable storage medium (Chen Pg 2 ¶08-¶09 discloses a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor), wherein the storage medium stores computer programs, which when executed by a processor (Chen Pg 2 ¶04-¶07 discloses an image enhancement device that provides a terminal device including a memory and processor where the memory is used to store a computer program that is then executed by the processor), cause the processor to:
obtain an initial image to be processed (Chen Pg 2 ¶02 discloses the initial image to be processed is and LDR image);
input the initial image into an image enhancement model (Chen Pg 1 ¶09 discloses inputting the image to be processed into a trained image enhancement model) obtained by pre-training (Chen Pg 5 ¶10 discloses the image enhancement model can be pre-trained by the image processing device), wherein the image enhancement model (Chen Pg 1 ¶09, Pg 2 ¶03, Pg 2 ¶07 discloses a trained image enhancement model) comprises a multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features);
perform a multi-scale feature extraction on an input image through the multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features) wherein the input image is obtained based on the initial image (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features or the initial image); and
obtain an image of which an image quality is enhanced (Chen Pg 1 ¶08-¶09 and Pg 2 ¶01 discloses the output being an enhanced image) and the initial image (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features).
Chen does not explicitly disclose to obtain initial feature maps of multiple scales, perform a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps, and perform a fusion based on the multiple intermediate feature maps to obtain an output feature map of the multi-scale feature fusion network, and based on the output feature map of the multi-scale feature fusion network.
Ma is in the same field of using feature fusion networks for image enhancement. Further, Ma teaches to obtain initial feature maps of multiple scales (Ma, Abstract and Section 2.3 ¶03 discloses feature maps at different scales of one image), perform a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps), and perform a fusion based on the multiple intermediate feature maps to obtain an output feature map of the multi-scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output),
and based on the output feature map of the multi-scale feature fusion network (Ma Fig 3 and Fig 4 and Section 2.3 and Section 2.4 discloses outputs F3-F5 which are intermediate feature maps and the functions included in the CBAM and convolution layers resulting in the four feature maps are subjected to element-wise summation and the final output feature map is the output).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen by incorporating the implementation of multiple scales into the network architecture to obtain multiple feature maps to create the multi scale feature fusion network as taught by Ma; to make an invention that can use the network to further refine the feature resolution of the images; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to address the key issues of remote sensing images by extracting the rich feature information of HR remote sensing images, better focusing on the change regions, avoiding the interference of other factors, and reducing the interference of pseudo-changes (Ma, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 21, Chen in view of Ma teaches the electronic device (Chen Pg 2 ¶04-¶07 discloses an image enhancement device that provides a terminal device including a memory and processor where the memory is used to store a computer program that is then executed by the processor Pg 9 ¶03 discloses electronic hardware) according to claim 18, wherein the performing a multi- scale feature extraction on an input image through the multi-scale feature fusion network (Chen Pg 2 ¶06 discloses the trained image enhancement model consisting of conditional network extracts multiple feature tensors of different scales from the image to be processed, and the enhanced image is obtained by feature fusion of the output of the main network with the original features) to obtain initial feature maps of multiple scales (Ma, Abstract and Section 2.3 ¶03 discloses feature maps at different scales of one image) comprises:
performing down-samplings on the input image according to a plurality of preset multiples (Chen Pg 1 ¶10 and Pg 4 ¶01 discloses M+1 feature extraction modules include different numbers of Downsampling operation, Pg 4 ¶02 discloses down sampling the initial image into three scales, the large scale, the intermediate scale and the small scale) respectively to obtain the initial feature maps of the multiple scales (Chen Pg 4 ¶02 discloses down sampling the initial image into three scales, the large scale, the intermediate scale and the small scale to obtain the spatial features), wherein the preset multiples are lower than a preset threshold (Chen Pg 4 ¶02 discloses down sampling the initial image into three scales, the large scale, the intermediate scale and the small scale to obtain the spatial features, the large scale is the original image size as disclosed by Chen, therefore the examiner is interpreting the original image size to be the threshold and all preset multiples are lower than the original as disclosed in Chen Pg 4 ¶02). See Claim 18 for rationale, its parent claim.
Claims 3-11, and 22 are rejected under 35 U.S.C. 103 as unpatentable over Chen in view of Ma in further view of Liu et al (Liu, Yu, et al. "Multiscale feature interactive network for multifocus image fusion." IEEE Transactions on Instrumentation and Measurement 70 (2021), hereafter referred to as Liu).
Regarding Claim 3, Chen in view of Ma teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 1, wherein the performing a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) comprises:
fusing the initial feature maps of the multiple scales (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) under different scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network) respectively to obtain an intermediate feature map (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) corresponding to each of the scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network), wherein intermediate feature maps (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) corresponding to each of the scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network).
Chen in view of Ma does not explicitly disclose have different spatial resolutions.
Liu is in the same field of using feature fusion networks for image enhancement. Further, Liu teaches have different spatial resolutions (Lu Pg 6 Col 2 ¶05 and Fig 1 discloses four focus maps of different resolutions).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen in view of Ma by incorporating different spatial resolutions and deep features to produce a deep feature map as taught by Lui; to make an invention that can use the network to further refine the feature resolution of the images; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to segment the focused and defocused regions of the source images accurately by sufficient interaction of multiscale features from layers of different depths, for multifocus image fusion. (Lui, Pg 2, Col 1 ¶03).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 4, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 3, wherein the fusing the initial feature maps of the multiple scales (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) under different scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network) respectively to obtain an intermediate feature map (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) corresponding to each of the scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network) comprises:
performing a fusion processing on the initial feature maps of the multiple scales (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) based on a self-attention mechanism to obtain a multi-scale fusion map (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps); and
taking the each of the scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network) as a target scale branch respectively (Ma Fig 2 discloses resizing to meet a target size, in this case the same as the input), and obtaining an intermediate feature map corresponding to the target scale branch (Ma Fig 2 discloses resizing to meet a target size, in this case the same as the input) based on the multi-scale fusion map (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) . See Claim 3 for rationale, its parent claim.
Regarding Claim 5, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 4, wherein the performing a fusion processing on the initial feature maps of the multiple scales (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) based on a self-attention mechanism to obtain a multi-scale fusion map (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) comprises:
unifying scales of the initial feature maps (Ma Fig 2 discloses resizing of the initial feature map to have the same size of the initial image) of the multiple scales to a scale (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network) corresponding to the target scale branch (Ma Fig 2 discloses resizing to meet a target size, in this case the same as the input), and performing an element-wise sum fusion on the initial feature maps after unifying scales (Ma Fig 2 discloses performing an element wise summation on the feature maps after the resizing) to obtain an initial fusion map (Ma Fig 2 discloses the output to be a feature map);
performing information compression (Ma Pg 9 ¶05 discloses preforming a compression mechanism) based on the initial fusion map (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) to obtain an information compression vector (Ma Pg 9 ¶05 discloses the compression on the channel dimension);
obtaining multiple feature vectors carrying attention information (Ma Pg 9 Section 2.3.1 discloses feature vectors obtained from the channel attention model) based on the information compression vector (Ma Pg 9 ¶05 discloses the compression on the channel dimension), wherein a number of the multiple feature vectors carrying the attention information (Ma Pg 9 Section 2.3.1 discloses feature vectors obtained from the channel attention model) is same with a number of scale types of the multiple scales (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network); and
performing a fusion processing (Ma Fig 3 discloses the fusion processing including the channel attention module where the feature vectors are made) according to the multiple feature vectors carrying the attention information (Ma Pg 9 Section 2.3.1 discloses feature vectors obtained from the channel attention model) to obtain the multi-scale fusion map (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps). See Claim 3 for rationale, its parent claim.
Regarding Claim 6, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 5, wherein the obtaining multiple feature vectors carrying attention information (Ma Pg 9 Section 2.3.1 discloses feature vectors obtained from the channel attention model) based on the information compression vector (Ma Pg 9 ¶05 discloses the compression on the channel dimension) comprises:
performing multiple convolutions (Ma Fig 1 discloses performing 5 convolutions) on the information compression vector (Ma Pg 9 ¶05 discloses the compression on the channel dimension) respectively to expand channels of the information compression vector to obtain multiple expanding feature vectors (Ma Fig 2 and Pg 7 ¶03 discloses a feature enhancement model that includes a resize function to expand the size of the vectors); and
performing a SoftMax activation (Liu Pg 5 Col 1 ¶05 discloses performing a SoftMax calculation to obtain the attention vectors) on the multiple extending feature vectors (Ma Fig 2 and Pg 7 ¶03 discloses a feature enhancement model that includes a resize function to expand the size of the vectors) respectively to obtain the multiple feature vectors carrying the attention information (Ma Pg 9 Section 2.3.1 discloses feature vectors obtained from the channel attention model). See Claim 3 for rationale, its parent claim.
Regarding Claim 7, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 4, wherein the obtaining an intermediate feature map corresponding to the target scale branch (Ma Fig 2 discloses resizing to meet a target size, in this case the same as the input) based on the multi-scale fusion map (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) comprises:
processing the multi-scale fusion map corresponding to the target scale branch (Ma Fig 2 discloses resizing to meet a target size, in this case the same as the input) based on the attention mechanism (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) to obtain the intermediate feature map corresponding to the target scale branch (Ma Fig 3 discloses the F4 feature map being an intermediate feature map and being the result of the channel attention module and spatial attention module). See Claim 3 for rationale, its parent claim.
Regarding Claim 8, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 7, wherein the processing the multi-scale fusion map corresponding to the target scale branch (Ma Fig 2 discloses resizing to meet a target size, in this case the same as the input) based on the attention mechanism (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) to obtain the intermediate feature map corresponding to the target scale branch (Ma Fig 3 discloses the F4 feature map being an intermediate feature map and being the result of the channel attention module and spatial attention module) comprises:
performing a deep feature extraction (Ma Pg 16 ¶01 discloses channel attention and spatial attention and uses a post-fusion strategy for deep supervision) on the multi-scale fusion map corresponding to the target scale branch (Ma Fig 2 discloses resizing to meet a target size, in this case the same as the input) to obtain a deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output);
processing the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output) based on a spatial attention mechanism to obtain a spatial attention feature map (Ma Fig 3 discloses using a spatial attention module to output the intermediate feature map);
processing the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output) based on a channel attention mechanism to obtain a channel attention vector (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps); and
performing a fusion processing (Ma Fig 3 discloses the fusion processing in the convolution block attention module) based on the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output), the spatial attention feature map (Ma Fig 3 discloses using a spatial attention module to output the intermediate feature map) and the channel attention vector (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) to obtain the intermediate feature map corresponding to the target scale branch (Ma Fig 3 discloses the F4 feature map being an intermediate feature map and being the result of the channel attention module and spatial attention module). See Claim 3 for rationale, its parent claim.
Regarding Claim 9, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 8, wherein the processing the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output) based on a spatial attention mechanism to obtain a spatial attention feature map (Ma Fig 3 discloses using a spatial attention module to output the intermediate feature map) comprises:
performing a Global Average Pooling (Li Pg 5 Col 1 ¶02 and Fig 2 discloses performing a global average pooling) on the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output) in a channel dimension to obtain a first feature map (Li Fig 2 and Pg 5 Col 1 ¶01 discloses performing the GAP to produce a feature map (orange box)), and performing a Global Max Pooling (Ma Pg 9 section 2.3.1 and 2.3.2 discloses max pooling) on the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output) in the channel dimension to obtain a second feature map (Li Fig 2 and Pg 5 Col 1 ¶01 discloses performing the GAP to produce a feature map (green box));
performing a cascade operation on the first feature map and the second feature map (Ma Fig 3 discloses performing an operation in series of the first and second feature map) to obtain a cascade feature map (Ma Fig 3 discloses performing an operation in series of the first and second feature map to obtain a third feature map); and
performing a dimension compression processing (Ma Pg 9 ¶05 discloses the compression on the channel dimension) and an activation processing (Ma Pg 9 Section 2.3.1 and 2.3.2 disclose sigmoid activation processing) on the cascade feature map (Ma Fig 3 discloses performing an operation in series of the first and second feature map to obtain a third feature map) to obtain the spatial attention feature map (Ma Fig 3 discloses the output of the spatial attention module to produce a F4 feature map). See Claim 3 for rationale, its parent claim.
Regarding Claim 10, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 8, wherein the performing a fusion processing (Ma Fig 3 discloses the fusion processing in the convolution block attention module) based on the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output), the spatial attention feature map (Ma Fig 3 discloses using a spatial attention module to output the intermediate feature map) and the channel attention vector (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) to obtain the intermediate feature map corresponding to the target scale branch (Ma Fig 3 discloses the F4 feature map being an intermediate feature map and being the result of the channel attention module and spatial attention module) comprises:
performing a point multiplication (Ma Fig 3 discloses performing elementwise multiplication in the spatial attention module to output a result) of the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output) and the spatial attention feature map (Ma Fig 3 discloses using a spatial attention module to output the intermediate feature map) to obtain a first point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the spatial attention module to output a result);
performing a point multiplication (Ma Fig 3 discloses performing elementwise multiplication in the channel attention module to output a result) of the deep feature map (Li Pg 16 Col 1 ¶06 and Fig 3 disclose the deep level features being the output) and the channel attention vector (Ma Fig 3 discloses using a channel attention module to output the intermediate feature map) to obtain a second point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the channel attention module to output a result); and
performing a fusion processing according to the first point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the spatial attention module to output a result) and the second point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the channel attention module to output a result) to obtain the intermediate feature map corresponding to the target scale branch (Ma Fig 3 discloses the F4 feature map being an intermediate feature map and being the result of the channel attention module and spatial attention module). See Claim 3 for rationale, its parent claim.
Regarding Claim 11, Chen in view of Ma in further view of Lui teaches the image enhancement method (Chen Pg 1 ¶03, ¶08 and Pg 2 ¶11 disclose an image enhancement method) according to claim 10, wherein the performing a fusion processing according to the first point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the spatial attention module to output a result) and the second point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the channel attention module to output a result) to obtain the intermediate feature map corresponding to the target scale branch (Ma Fig 3 discloses the F4 feature map being an intermediate feature map and being the result of the channel attention module and spatial attention module) comprises:
Cascading (Ma Fig 3 discloses that the element wise point multiplication are performed in a sequence) the first point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the spatial attention module to output a result) and the second point multiplication result (Ma Fig 3 discloses performing elementwise multiplication in the channel attention module to output a result) to obtain a two-channel feature map (Ma Pg 9 Section 2.3.2 discloses a 2-channel feature map);
performing a convolution (Ma Pg 9 Section 2.4 discloses performing convolutions to match the input feature map) on the two-channel feature map (Ma Pg 9 Section 2.3.2 discloses a 2-channel feature map) to obtain a one-channel feature map (Ma Pg 9 Section 2.4 discloses performing convolutions to match the input feature map); and
adding the one-channel feature map (Ma Pg 9 Section 2.4 discloses performing convolutions to match the input feature map) and the multi-scale fusion map corresponding to the target scale branch (Ma Pg 3 ¶01 discloses using a self-attention module and Fig 3 discloses a channel attention module for using in creating the fusion maps) to obtain the intermediate feature map corresponding to the target scale branch (Ma Fig 3 discloses the F4 feature map being an intermediate feature map and being the result of the channel attention module and spatial attention module). See Claim 3 for rationale, its parent claim.
Regarding Claim 22, Chen in view of Ma teaches the electronic device (Chen Pg 2 ¶04-¶07 discloses an image enhancement device that provides a terminal device including a memory and processor where the memory is used to store a computer program that is then executed by the processor Pg 9 ¶03 discloses electronic hardware) according to claim 18, wherein the performing a fusion based on the initial feature maps of the multiple scales to obtain multiple intermediate feature maps (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) comprises:
fusing the initial feature maps of the multiple scales (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) under different scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network) respectively to obtain an intermediate feature map (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) corresponding to each of the scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network), wherein intermediate feature maps (Ma Fig 3 and Section 2.3 discloses a fusion based on the initial feature maps which are represented by F1 and F2, which outputs F3-F5 which are intermediate feature maps) corresponding to each of the scale branches (Chen Pg 2 ¶06 discloses multiple feature tensors of different scales to the network layer of the corresponding scale in the main network).
Chen in view of Ma does not explicitly disclose have different spatial resolutions.
Liu is in the same field of using feature fusion networks for image enhancement. Further, Liu teaches have different spatial resolutions (Lu Pg 6 Col 2 ¶05 and Fig 1 discloses four focus maps of different resolutions).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Chen in view of Ma by incorporating different spatial resolutions and deep features to produce a deep feature map as taught by Lui; to make an invention that can use the network to further refine the feature resolution of the images; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to segment the focused and defocused regions of the source images accurately by sufficient interaction of multiscale features from layers of different depths, for multifocus image fusion. (Lui, Pg 2, Col 1 ¶03).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Reference Cited
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
CN-113902903-A to Zhang et al. discloses a dual attention multi-scale fusion method based on down sampling.
CN-111915613-A to Du et al. discloses a segmentation technique for images to segment out instances of interest.
Cui, Heng Shuai, et al. "Attention-guided multi-scale feature fusion network for low-light image enhancement." Frontiers in neurorobotics 16 (2022) discloses an Attention-Guided Multi-scale feature fusion network for low light image enhancement based on the structure of a single encoder and decoder and the principle of coarse-to-fine network design.
Wang, Xinying, et al. "Remote sensing imagery super resolution based on adaptive multi-scale feature fusion network." Sensors 20.4 (2020) discloses an adaptive multi-scale feature fusion network (AMFFN) for remote sensing image super-resolution.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL ROBERTS whose telephone number is (571)272-6413. The examiner can normally be reached Monday- Friday 7:30am- 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on (313) 446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL L ROBERTS/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights