DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record on file.
Information Disclosure Statement(s)
The Information disclosure statement (IDS) filed on October 31st, 2024 has been acknowledged and considered by the examiner.
Drawings
Figure 6 is objected to as depicting a block diagram without “readily identifiable” descriptors of each block, as required by 37 CFR 1.84(n). Rule 84(n) requires “labeled representations” of graphical symbols, such as blocks; and any that are “not universally recognized may be used, subject to approval by the Office, if they are not likely to be confused with existing conventional symbols, and if they are readily identifiable.” In the case of figure 6, the blocks are not readily identifiable per se and therefore require the insertion of text that identifies the function of that block. That is, each vacant block should be provided with a corresponding label identifying its function or purpose.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 2, 7-8, 10, 17, 19 and 20 are objected to because of the following informalities:
Claim 2, line 1, “wherein the generating of the temporal feature representations” should be read as “wherein the generating inter-image temporal feature representations”, to follow proper antecedent basis, since there is no prior instantiation of such antecedent reference to have such antecedent referencing here. Appropriate correction is required to avoid 112(b) antecedent basis issue.
Claim 7, line 1, “the individual image frames” should be read as “individual image frames of the respective image frames”, since “the individual image frames” is lack of antecedent basis, since there is no prior instantiation of such antecedent reference to have such antecedent referencing here. Appropriate correction is required to avoid 112(b) antecedent basis issue.
Claim 8, line 1, “the individual image frames” should be read as “individual image frames of the respective image frames”, since “the individual image frames” is lack of antecedent basis, since there is no prior instantiation of such antecedent reference to have such antecedent referencing here. Appropriate correction is required to avoid 112(b) antecedent basis issue.
Claim 10, line 5, “image frames .” should be read as “image frames.” to follow proper claim language and grammar, no space between word and a period “.”. Appropriate correction is required.
Claim 17, line 1, “the individual image frames” should be read as “individual image frames of the respective image frames”, since “the individual image frames” is lack of antecedent basis, since there is no prior instantiation of such antecedent reference to have such antecedent referencing here. Appropriate correction is required to avoid 112(b) antecedent basis issue.
Claim 19, line 2, “the individual image frames” should be read as “individual image frames of the respective image frames”, since “the individual image frames” is lack of antecedent basis, since there is no prior instantiation of such antecedent reference to have such antecedent referencing here. Appropriate correction is required to avoid 112(b) antecedent basis issue.
Claim 20, lines 4-5, “the frame images” should be read as “image frames of the respective image frames”, since “the frame images” is lack of antecedent basis, since there is no prior instantiation of such antecedent reference to have such antecedent referencing here. Appropriate correction is required to avoid 112(b) antecedent basis issue.
Claim 20, line 6, “the individual image frames” should be read as “individual image frames of the respective image frames”, since “the individual image frames” is lack of antecedent basis, since there is no prior instantiation of such antecedent reference to have such antecedent referencing here. Appropriate correction is required to avoid 112(b) antecedent basis issue.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitation(s) that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function.
Claims 1 and 11 , recite(s) limitation(s) that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 1; recites the limitation, “extracting, by a spatial feature extractor, intra-image…,” [Line 2].
Claim 11; recites the limitation, “extract, by a spatial feature extractor, intra-image…,” [Line 4].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 1 and 11;
(i) “a spatial feature extractor”: Paragraph [0053] of the instant specification, filed on October 31st, 2024, wherein it discloses the spatial feature extractor may be/include a neural network capable of feature extraction, such as a CNN or a transformer thus have sufficient structure or material/act wherein is a feature extraction neural network of a CNN or a transformer).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 6-9, 11, 14 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Xiaoou Tang et. al. (“US 2021/0241470 A1” hereinafter as “Tang”) in view of Asim Kadav et. al. (“US 2022/0101007 A1” hereinafter as “Kadav”).
(best understood based on the 112f interpretation above) Regarding claim 1, Tang explicitly teaches an image enhancement method comprising (Par. [0069] discloses “through the method…a high-quality frame may usually be restored, and image restoration is realized”): extracting intra-image spatial feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of an intra-frame spatial relationship for multi-frame fusion for aligning feature data indicating an extracting of the intra-frame spatial feature representations [intra-frame spatial feature data since it includes multi frames]) from respective image frames of a burst image set (Par. [0061] discloses “for multi-frame fusion. Because different adjacent frames have different amounts of information due to problems of occlusion” indicating from different frames of a burst image set [images with occlusion]); generating inter-image temporal feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of inter-frame temporal feature representations [inter-frame temporal relationship and feature data, and of a multi frames indicating a plurality of representations]) based on a local similarity between the spatial feature representations (Par. [0061] discloses “temporal attention is to calculate a similarity between frames embedded in a space” indicating the temporal features is based on similarity between frames; moreover, Par. [0138] discloses “the temporal attention mechanism may endow information of different regions of different frames with different information” indicating the information being used for the temporal attention being spatial information, therefore, the similarity calculation, as discussed, can be understood to be based on spatial information/spatial feature representations of the frames); determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating a fusing of the spatial feature representations with the temporal feature representation, and the resulted fused feature data can be understood to be analogous to the recited temporal-spatial feature representations, Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion” indicating a temporal and spatial attention mechanism based on the temporal and spatial multi-frame fusion, the temporal and spatial attention here being analogous to the temporal-spatial feature representations as claimed); selecting a base image frame from among the image frames (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]) based on the temporal-spatial feature representations (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame is based on the fused feature data being the temporal-spatial feature representations; Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” and Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion” indicating a temporal and spatial attention mechanism based on the temporal and spatial multi-frame fusion, the temporal and spatial attention here being analogous to the temporal-spatial feature representations as claimed); and generating an enhanced image by performing an image enhancement operation (Par. [0069] discloses “the fused information of the image frame sequence can be obtained, and image reconstruction may further be performed according to the fused information to obtain the processed image frame corresponding to the image frame to be processed. A high-quality frame may usually be restored, and image restoration is realized”) on the burst image set ((Par. [0061] discloses “for multi-frame fusion. Because different adjacent frames have different amounts of information due to problems of occlusion” indicating from different frames of a burst image set [images with occlusion]) based on the base image frame (Par. [0131] discloses “then the fused information is input to a reconstruction module to acquire processed image frames according to the fused information, and an up-sampling operation is executed at the end of the network to enlarge a space size. Finally a predicted image residual is added to an image obtained by directly up-sampling the original image frame, so that a high-resolution frame may be obtained” indicating a further image enhancement process is performed according the processed images [including the base image frame as discussed]).
However, Tang does not explicitly teach extracting, by a spatial feature extractor, intra-image spatial feature representations.
In the same field of spatial-temporal feature extraction from frames (Title and Abstract, Kadav), Kadav explicitly teaches extracting, by a spatial feature extractor, intra-image spatial feature representations (Par. [0026] discloses “Hopper obtains representations for the spatial context and every frame via the backbone of CNN” indicating extracting of spatial feature representations for frames [spatial context for every frame] being intra-image [for every frame] using a CNN [spatial feature extractor as claimed]; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to have a process of extracting intra-frame spatial feature representation from images, wherein the extraction can be done by a CNN model. Thus in order to have use Hopper CNN model to obtain spatial context of images in a great accuracy, see Par. [0062] of Kadav).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement method comprising: extracting intra-image spatial feature representations from respective image frames of a burst image set;. Moreover, Tang extraction of intra-frame spatial feature representation can be modified to be based on the use of a CNN as taught in Kadav.
Such a modification is the result of combing prior art elements. Tang and Kadav share the same field of endeavor of spatial-temporal feature extraction from frames. The motivation for the proposed modification would have been to have an image enhancement method comprising: extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set in order to have use Hopper CNN model to obtain spatial context of images in a great accuracy, see Par. [0062] of Kadav.
Regarding claim 4, Tang in view of Kadav, in combination, explicitly teaches the image enhancement method of claim 1, wherein Tang explicitly teaches the temporal feature representations comprise: motion information of the respective image frames (Par. [0137] discloses “the alignment module for multi-frame alignment…complex motion or a motion”; moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data”, indicating that the multi-frame fusion/alignment is based on motion feature data, which can be understood that the temporal feature representations comprise motion information).
Regarding claim 6, Tang in view of Kadav, in combination, explicitly teaches the image enhancement method of claim 1, wherein Tang explicitly teaches the generating of the temporal feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of inter-frame temporal feature representations [inter-frame temporal relationship and feature data, and of a multi frames indicating a plurality of representations]) and the determining of the temporal-spatial feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating a fusing of the spatial feature representations with the temporal feature representation, and the resulted fused feature data can be understood to be analogous to the recited temporal-spatial feature representations, Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion”) are iteratively performed (Par. [0131] discloses “multi-frame alignment and fusion is performed on each frame with the adjacent frames, to finally obtain fused information” indicating the multi-frame fusion is performed iteratively on each frame to the adjacent frames), and the base image frame is selected (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]) based on final temporal-spatial feature representations (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame is based on the fused feature data being the temporal-spatial feature representations) obtained as a result of the iterations (Par. [0131] discloses “multi-frame alignment and fusion is performed on each frame…to finally obtain fused information…to acquire processed image frame” being the base frame being selected as the result).
Regarding claim 7, Tang in view of Kadav, in combination, explicitly teaches the image enhancement method of claim 1, wherein Tang explicitly teaches the individual image frames have composite degradation (Par. [0129] discloses “training dataset may include a pair of…blurred and non-blurred sample image frame” indicating a degradation, and pair of images indicating a composite of the degradation).
Regarding claim 8, Tang in view of Kadav, in combination, explicitly teaches the image enhancement method of claim 1, wherein Tang explicitly teaches the individual image frames are captured with different respective exposure times (Par. [0032] discloses “continuous frames of images may form a video…a frame rate generally refers to a frame number of pictures transmitted in one second, and may be understood as a number of refresh times that a graphics processing unit can implement in each second” indicating that the image frames being captured at different time intervals according to the frame rate, therefore, being analogous to different respective exposure times as claimed).
Regarding claim 9, Tang in view of Kadav, in combination, explicitly teaches the image enhancement method of claim 1, wherein Tang explicitly teaches the selecting of the base image frame (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]) comprises: generating a selection guide vector corresponding to the temporal-spatial feature representations (Par. [0059] discloses “for any two pieces of aligned feature data, the weight information may be calculated by means of a dot product of vectors” therefore, the aligned feature data [associated fused feature data resulted product of vectors is analogous to the selection guide vector as claimed] since, the weight value [product vectors] is used to indicate the aligned feature data, Par. [0059] discloses “the weight information may be calculated by means of a dot product of vectors…if a weight value is higher, it is usually indicated that the aligned feature data is more important among all the frames” which is used to select the base image, Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]), wherein elements of the selection guide vector respectively correspond to the frame images (Par. [0097] discloses “the weight information of each of the plurality of pieces of aligned feature data is determined by a preset activation function and the plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed” indicating the elements of the weight information [selection guide vector] correspond to the frame images); and selecting the base image frame from among the individual image frames based on values of the elements of the selection guide vector (Par. [0097] discloses “the weight information of each of the plurality of pieces of aligned feature data is determined by a preset activation function and the plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed” indicating the base image frame is selected based on the selection guide vector [the weight information as discussed]).
(best understood based on the 112f interpretation above) Regarding claim 11, Tang explicitly teaches an electronic device comprising: one or more processors; and a memory storing instructions configured to cause the one or more processors to: (Par. [0069] discloses “through the method…a high-quality frame may usually be restored, and image restoration is realized”; moreover, Par. [0008] discloses “an electronic device, including a processor and a memory. The memory is configured to store instructions which, when being executed by the processor, cause the processor to carry out the following…”): extract intra-image spatial feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of an intra-frame spatial relationship for multi-frame fusion for aligning feature data indicating an extracting of the intra-frame spatial feature representations [intra-frame spatial feature data since it includes multi frames]) from respective image frames of a burst image set (Par. [0061] discloses “for multi-frame fusion. Because different adjacent frames have different amounts of information due to problems of occlusion” indicating from different frames of a burst image set [images with occlusion]); generate inter-image temporal feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of inter-frame temporal feature representations [inter-frame temporal relationship and feature data, and of a multi frames indicating a plurality of representations]) based on a local similarity between the spatial feature representations (Par. [0061] discloses “temporal attention is to calculate a similarity between frames embedded in a space” indicating the temporal features is based on similarity between frames; moreover, Par. [0138] discloses “the temporal attention mechanism may endow information of different regions of different frames with different information” indicating the information being used for the temporal attention being spatial information, therefore, the similarity calculation, as discussed, can be understood to be based on spatial information/spatial feature representations of the frames); determine temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating a fusing of the spatial feature representations with the temporal feature representation, and the resulted fused feature data can be understood to be analogous to the recited temporal-spatial feature representations, Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion” indicating a temporal and spatial attention mechanism based on the temporal and spatial multi-frame fusion, the temporal and spatial attention here being analogous to the temporal-spatial feature representations as claimed); select a base image frame from among the image frames (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]) based on the temporal-spatial feature representations (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame is based on the fused feature data being the temporal-spatial feature representations; Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” and Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion” indicating a temporal and spatial attention mechanism based on the temporal and spatial multi-frame fusion, the temporal and spatial attention here being analogous to the temporal-spatial feature representations as claimed); and generate an enhanced image by performing an image enhancement operation (Par. [0069] discloses “the fused information of the image frame sequence can be obtained, and image reconstruction may further be performed according to the fused information to obtain the processed image frame corresponding to the image frame to be processed. A high-quality frame may usually be restored, and image restoration is realized”) on the burst image set ((Par. [0061] discloses “for multi-frame fusion. Because different adjacent frames have different amounts of information due to problems of occlusion” indicating from different frames of a burst image set [images with occlusion]) based on the base image frame (Par. [0131] discloses “then the fused information is input to a reconstruction module to acquire processed image frames according to the fused information, and an up-sampling operation is executed at the end of the network to enlarge a space size. Finally a predicted image residual is added to an image obtained by directly up-sampling the original image frame, so that a high-resolution frame may be obtained” indicating a further image enhancement process is performed according the processed images [including the base image frame as discussed]).
However, Tang does not explicitly teach extract, by a spatial feature extractor, intra-image spatial feature representations.
In the same field of spatial-temporal feature extraction from frames (Title and Abstract, Kadav), Kadav explicitly teaches extract, by a spatial feature extractor, intra-image spatial feature representations (Par. [0026] discloses “Hopper obtains representations for the spatial context and every frame via the backbone of CNN” indicating extracting of spatial feature representations for frames [spatial context for every frame] being intra-image [for every frame] using a CNN [spatial feature extractor as claimed]; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to have a process of extracting intra-frame spatial feature representation from images, wherein the extraction can be done by a CNN model. Thus in order to have use Hopper CNN model to obtain spatial context of images in a great accuracy, see Par. [0062] of Kadav).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement device comprising extracting intra-image spatial feature representations from respective image frames of a burst image set;. Moreover, Tang extraction of intra-frame spatial feature representation can be modified to be based on the use of a CNN as taught in Kadav.
Such a modification is the result of combing prior art elements. Tang and Kadav share the same field of endeavor of spatial-temporal feature extraction from frames. The motivation for the proposed modification would have been to have an image enhancement device comprising extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set in order to have use Hopper CNN model to obtain spatial context of images in a great accuracy, see Par. [0062] of Kadav.
Regarding claim 14, Tang in view of Kadav, in combination, explicitly teaches the electronic device of claim 11, wherein Tang explicitly teaches the temporal feature representations comprise: motion information of the respective image frames (Par. [0137] discloses “the alignment module for multi-frame alignment…complex motion or a motion”; moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data”, indicating that the multi-frame fusion/alignment is based on motion feature data, which can be understood that the temporal feature representations comprise motion information).
Regarding claim 16, Tang in view of Kadav, in combination, explicitly teaches the electronic device of claim 11, wherein Tang explicitly teaches the generating of the temporal feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of inter-frame temporal feature representations [inter-frame temporal relationship and feature data, and of a multi frames indicating a plurality of representations]) and the determining of the temporal-spatial feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating a fusing of the spatial feature representations with the temporal feature representation, and the resulted fused feature data can be understood to be analogous to the recited temporal-spatial feature representations, Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion”) are iteratively performed (Par. [0131] discloses “multi-frame alignment and fusion is performed on each frame with the adjacent frames, to finally obtain fused information” indicating the multi-frame fusion is performed iteratively on each frame to the adjacent frames), and the base image frame is selected (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]) based on final temporal-spatial feature representations (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame is based on the fused feature data being the temporal-spatial feature representations) obtained as a result of the iterations (Par. [0131] discloses “multi-frame alignment and fusion is performed on each frame…to finally obtain fused information…to acquire processed image frame” being the base frame being selected as the result).
Regarding claim 17, Tang in view of Kadav, in combination, explicitly teaches the electronic device of claim 11, wherein Tang explicitly teaches the individual image frames have composite degradation (Par. [0129] discloses “training dataset may include a pair of…blurred and non-blurred sample image frame” indicating a degradation, and pair of images indicating a composite of the degradation).
Regarding claim 18, Tang in view of Kadav, in combination, explicitly teaches the electronic device of claim 11, wherein Tang explicitly teaches wherein the electronic device further comprises: a camera configured to capture the burst image set (Par. [0032] discloses “continuous frames of images may form a video…a frame rate generally refers to a frame number of pictures transmitted in one second, and may be understood as a number of refresh times that a graphics processing unit can implement in each second” indicating that the image frames being captured at different time intervals according to the frame rate, therefore, being analogous to different respective exposure times as claimed, indicating the use of a camera to capture the image set).
Regarding claim 19, Tang in view of Kadav, in combination, explicitly teaches The electronic device of claim 11, wherein Tang explicitly teaches wherein the electronic device further comprises: a camera configured to capture the burst image set (Par. [0032] discloses “continuous frames of images may form a video…a frame rate generally refers to a frame number of pictures transmitted in one second, and may be understood as a number of refresh times that a graphics processing unit can implement in each second” indicating that the image frames being captured at different time intervals according to the frame rate, therefore, being analogous to different respective exposure times as claimed, indicating the use of a camera to capture the image set).
Regarding claim 20, Tang in view of Kadav, in combination, explicitly teaches The electronic device of claim 11, wherein Tang explicitly teaches wherein, in order to select the base image frame, the instructions are further configured to cause the one or more processors to: (Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]) generate a selection guide vector corresponding to the temporal-spatial feature representations (Par. [0059] discloses “for any two pieces of aligned feature data, the weight information may be calculated by means of a dot product of vectors” therefore, the aligned feature data [associated fused feature data resulted product of vectors is analogous to the selection guide vector as claimed] since, the weight value [product vectors] is used to indicate the aligned feature data, Par. [0059] discloses “the weight information may be calculated by means of a dot product of vectors…if a weight value is higher, it is usually indicated that the aligned feature data is more important among all the frames” which is used to select the base image, Par. [0063] discloses “the plurality of pieces of aligned feature data are fused according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence. The fused information is configured to acquire a processed image frame corresponding to the image frame to be processed” indicating a processes image frame to be resulted [selecting a base image as claimed]), wherein elements of the selection guide vector respectively correspond to the frame images (Par. [0097] discloses “the weight information of each of the plurality of pieces of aligned feature data is determined by a preset activation function and the plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed” indicating the elements of the weight information [selection guide vector] correspond to the frame images); and select the base image frame from among the individual image frames based on values of the elements of the selection guide vector (Par. [0097] discloses “the weight information of each of the plurality of pieces of aligned feature data is determined by a preset activation function and the plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed” indicating the base image frame is selected based on the selection guide vector [the weight information as discussed]).
Claims 2-3 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Xiaoou Tang et. al. (“US 2021/0241470 A1” hereinafter as “Tang”) in view of Asim Kadav et. al. (“US 2022/0101007 A1” hereinafter as “Kadav”) and William Wright et. al. (“US 8,966,398 B2” hereinafter as “Wright”).
Regarding claim 2, Tang in view of Kadav, in combination, explicitly teaches the image enhancement method of claim 1, wherein Tang explicitly teaches the generating of the temporal feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of inter-frame temporal feature representations [inter-frame temporal relationship and feature data, and of a multi frames indicating a plurality of representations]).
However, Tang in view of Kadav does not explicitly teach the generating of the temporal feature representations comprises: selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations.
In the same field of generating temporal-spatial feature data (Title and Abstract, Wright), Wright explicitly teaches the generating of the temporal feature representations comprises: selecting a target spatial feature representation from among the spatial feature representations (Col. 12, lines 32-60, discloses “Spatial Domain Representation…organizing element of the visualization representation is the 2D/3D spatial reference frame” which is analogous to selecting a target spatial feature representation [spatial reference frame] among the plurality of spatial frames); comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations (Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating spatially-corresponding search regions [different spatial locations is synchronized]); and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison (Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time”; moreover, Col. 17, lines 10-15, discloses “in other words, the time scale is the same across all timelines in the time domain of the visual representation. Therefore, it is recognized that the timelines are used in the visual representation to visually depict a graphical visualization of the data objects over time with respect to their spatial properties/attributes” indicating generating temporal feature representation corresponding to the target spatial representation as the result of the comparison as claimed), the temporal feature representation being included in the temporal feature representations (Col. 17, lines 16-31, discloses “the time range represented by the timelines can be synchronized. In other words, the time scale can be selected as the same for every timeline of the selected time range of the temporal domain of the representation” indicating the selected temporal feature is part of the temporal domain including the temporal feature representations as claimed; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to generate inter-image temporal feature representations based on a local similarity between the spatial feature representations, wherein the generation can be modified to be based on selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations. Thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement method comprising: extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set; generating inter-image temporal feature representations based on a local similarity between the spatial feature representations; determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations. Moreover, Tang’s wherein the generation can be modified to be based on selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations as taught in Wright.
Such a modification is the result of combing prior art elements. The motivation for the proposed modification would have been to have an image enhancement method comprising extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set; generating inter-image temporal feature representations based on a local similarity between the spatial feature representations; determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations; wherein the generation can be modified to be based on selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations, thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55.
Regarding claim 3, Tang in view of Kadav and Wright, in combination, explicitly teaches the image enhancement method of claim 2.
However, Tang in view of Kadav in combination does not explicitly teach wherein the comparing of the window regions of the target spatial feature representation with the spatially-corresponding search regions of the spatial feature representations comprises: selecting a first spatial feature representation from among the spatial feature representations; comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation.
In the same field of generating temporal-spatial feature data (Title and Abstract, Wright), Wright explicitly teaches wherein the comparing of the window regions of the target spatial feature representation with the spatially-corresponding search regions of the spatial feature representations (Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating spatially-corresponding search regions [different spatial locations is synchronized]) comprises: selecting a first spatial feature representation from among the spatial feature representations (Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating the comparison is events and sequences between locations indicating selecting locations and associated events [analogous to selecting a first spatial feature representation from among the spatial feature representations as claimed]); comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation (Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating the comparison is events and sequences between locations indicating selecting locations and associated events; Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation, any of which is analogous to the recited first window region] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, indicating spatially-corresponding search regions [different spatial locations is synchronized, any of which is analogous to the recited first spatially-corresponding search region]); and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation (Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating the comparison is events and sequences between locations indicating selecting locations and associated events; Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation, any of which, other than the previously selected, is analogous to the recited second window region] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, indicating spatially-corresponding search regions [different spatial locations is synchronized, any of which, other than the previously selected, is analogous to the recited second spatially-corresponding search region] ; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to generate inter-image temporal feature representations based on a local similarity between the spatial feature representations by comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations, wherein the comparison can be modified to be based on selecting a first spatial feature representation from among the spatial feature representations; comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation. Thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement method comprising generating inter-image temporal feature representations based on a local similarity between the spatial feature representations by comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations. Moreover, Tang’s wherein the generation can be modified to be based on selecting a first spatial feature representation from among the spatial feature representations; comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation as taught in Wright.
Such a modification is the result of combing prior art elements. The motivation for the proposed modification would have been to generate inter-image temporal feature representations based on a local similarity between the spatial feature representations by comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations, wherein the comparison can be modified to be based on selecting a first spatial feature representation from among the spatial feature representations; comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation, thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55.
Regarding claim 12, Tang in view of Kadav, in combination, explicitly teaches the electronic device of claim 11, wherein Tang explicitly teaches in order to generate the temporal feature representations, the instructions are further configured to cause the one or more processors to: (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating the use of inter-frame temporal feature representations [inter-frame temporal relationship and feature data, and of a multi frames indicating a plurality of representations]).
However, Tang in view of Kadav does not explicitly teach select a target spatial feature representation from among the spatial feature representations; compare window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generate a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations.
In the same field of generating temporal-spatial feature data (Title and Abstract, Wright), Wright explicitly teaches select a target spatial feature representation from among the spatial feature representations (Col. 12, lines 32-60, discloses “Spatial Domain Representation…organizing element of the visualization representation is the 2D/3D spatial reference frame” which is analogous to selecting a target spatial feature representation [spatial reference frame] among the plurality of spatial frames); compare window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations (Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating spatially-corresponding search regions [different spatial locations is synchronized]); and generate a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison (Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time”; moreover, Col. 17, lines 10-15, discloses “in other words, the time scale is the same across all timelines in the time domain of the visual representation. Therefore, it is recognized that the timelines are used in the visual representation to visually depict a graphical visualization of the data objects over time with respect to their spatial properties/attributes” indicating generating temporal feature representation corresponding to the target spatial representation as the result of the comparison as claimed), the temporal feature representation being included in the temporal feature representations (Col. 17, lines 16-31, discloses “the time range represented by the timelines can be synchronized. In other words, the time scale can be selected as the same for every timeline of the selected time range of the temporal domain of the representation” indicating the selected temporal feature is part of the temporal domain including the temporal feature representations as claimed; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to generate inter-image temporal feature representations based on a local similarity between the spatial feature representations, wherein the generation can be modified to be based on selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations. Thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement device comprising: extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set; generating inter-image temporal feature representations based on a local similarity between the spatial feature representations; determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations. Moreover, Tang’s wherein the generation can be modified to be based on selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations as taught in Wright.
Such a modification is the result of combing prior art elements. The motivation for the proposed modification would have been to have an image enhancement device comprising extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set; generating inter-image temporal feature representations based on a local similarity between the spatial feature representations; determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations; wherein the generation can be modified to be based on selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations; and generating a temporal feature representation corresponding to the target spatial feature representation based on a result of the comparison, the temporal feature representation being included in the temporal feature representations, thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55.
Regarding claim 13, Tang in view of Kadav and Wright, in combination, explicitly teaches the electronic device of claim 12.
However, Tang in view of Kadav in combination does not explicitly teach in order to compare the window regions of the target spatial feature representation with the search regions of the spatial feature representations, the instructions are further configured to cause the one or more processors to: select a first spatial feature representation from among the spatial feature representations; compare a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and compare a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation.
In the same field of generating temporal-spatial feature data (Title and Abstract, Wright), Wright explicitly teaches in order to compare the window regions of the target spatial feature representation with the search regions of the spatial feature representations, the instructions are further configured to cause the one or more processors to: (Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating spatially-corresponding search regions [different spatial locations is synchronized]) select a first spatial feature representation from among the spatial feature representations (Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating the comparison is events and sequences between locations indicating selecting locations and associated events [analogous to selecting a first spatial feature representation from among the spatial feature representations as claimed]); compare a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation (Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating the comparison is events and sequences between locations indicating selecting locations and associated events; Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation, any of which is analogous to the recited first window region] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, indicating spatially-corresponding search regions [different spatial locations is synchronized, any of which is analogous to the recited first spatially-corresponding search region]); and compare a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation (Col. 17, lines 1-10, discloses “a single spatial view will have as many time-lines as necessary…in order to make comparisons between events and sequences of event between locations, the time range represented by multiple times lines projecting through the reference surface at different spatial locations is synchronized” indicating the comparison is events and sequences between locations indicating selecting locations and associated events; Col. 12, lines 45-60, discloses “navigate the reference surface by scrolling in any direction, zooming in or out of an area and selecting specific areas of focus. In this way…can specify the spatial dimensions of an area of interest the reference surface in which to view events in time” indicating a zooming in and out of selecting areas of focus in the spatial reference frame [window regions of the target spatial feature representation, any of which, other than the previously selected, is analogous to the recited second window region] to correspond to events in time of that area [using a reference spatial frame to find the corresponding events in other frames in time], moreover, the correspondence is based on comparisons; Col. 17, lines 1-10, indicating spatially-corresponding search regions [different spatial locations is synchronized, any of which, other than the previously selected, is analogous to the recited second spatially-corresponding search region] ; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to generate inter-image temporal feature representations based on a local similarity between the spatial feature representations by comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations, wherein the comparison can be modified to be based on selecting a first spatial feature representation from among the spatial feature representations; comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation. Thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement device comprising generating inter-image temporal feature representations based on a local similarity between the spatial feature representations by comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations. Moreover, Tang’s wherein the generation can be modified to be based on selecting a first spatial feature representation from among the spatial feature representations; comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation as taught in Wright.
Such a modification is the result of combing prior art elements. The motivation for the proposed modification would have been to generate inter-image temporal feature representations based on a local similarity between the spatial feature representations by comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations, wherein the comparison can be modified to be based on selecting a first spatial feature representation from among the spatial feature representations; comparing a first window region of the window regions of the target spatial feature representation with a first search region spatially-corresponding to the first window region in the first spatial feature representation; and comparing a second window region of the window regions of the target spatial feature representation with a second search region spatially-corresponding to the second window region in the first spatial feature representation, thus in order to have improved perception of entity activities using such method, see Wright’s Col. 10, lines 42-55.
Claims 5, 10 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Xiaoou Tang et. al. (“US 2021/0241470 A1” hereinafter as “Tang”) in view of Asim Kadav et. al. (“US 2022/0101007 A1” hereinafter as “Kadav”) and Minsu Cho et. al. (“US 2024/0153238 A1” hereinafter as “Cho”).
Regarding claim 5, Tang in view of Kadav, in combination, explicitly teaches the image enhancement method of claim 1, wherein Tang explicitly teaches the determining of the temporal-spatial feature representations (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating a fusing of the spatial feature representations with the temporal feature representation, and the resulted fused feature data can be understood to be analogous to the recited temporal-spatial feature representations, Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion”).
However, Tang in view of Kadav, in combination, does not explicitly teach the determining of the temporal-spatial feature representations comprises: causing a size of the spatial feature representations to equal a size of the temporal feature representations; and determining the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations.
In the same field of spatial-temporal feature extraction (Title and Abstract, Cho), Cho explicitly teaches the determining of the temporal-spatial feature representations comprises: causing a size of the spatial feature representations to equal a size of the temporal feature representations (Par. [0109] discloses “the STSS feature map may have the same size as that of the video feature map”; moreover, Par. [0106] discloses “a spatial self-similarity map and a plurality of spatial cross-similarity maps for each position in a video feature map” indicating the video feature map include spatial feature representations; and Par. STSS feature map [spatial-temporal self-similarity feature map] includes temporal feature representations, therefore, it can be understood that the size of the spatial feature representations is equal to a size of the temporal feature representations, as claimed); and determining the temporal-spatial feature representations based on an elementwise addition operation (Par. [0109] discloses “the STSS feature map may have the same size as that of the video feature map…the electronic device may add the STSS feature map and the video feature map through an elementwise addition”; moreover, Par. [0102] discloses “the calculated final STSS feature map Z may be added, elementwise, to the video feature map V, and thus the SELFY block may operate as residual block for motion learning” indicating a result of offset-applied STSS feature map [analogous to the temporal-spatial feature representations as claimed being determined]) of feature values of the spatial feature representations and feature values of the temporal feature representations (Par. [0109] discloses “the STSS feature map may have the same size as that of the video feature map…the electronic device may add the STSS feature map and the video feature map through an elementwise addition”; moreover, Par. [0102] discloses “the calculated final STSS feature map Z may be added, elementwise, to the video feature map V, and thus the SELFY block may operate as residual block for motion learning” indicating the feature values of the spatial feature representations and feature values of the temporal feature representations; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to have a process of determining of the temporal-spatial feature representations can be modified to be based on causing a size of the spatial feature representations to equal a size of the temporal feature representations; and determining the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations. Thus in order to use a SELFY block to perform the method as discussed to improve the video representation ability of a video-processing artificial neural network to perform the processing more effectively, see Cho, Par. [0069]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement method comprising extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set; generating inter-image temporal feature representations based on a local similarity between the spatial feature representations; determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations. Moreover, Tang’s determining of the temporal-spatial feature representations can be modified to be based on causing a size of the spatial feature representations to equal a size of the temporal feature representations; and determining the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations as taught in Cho.
Such a modification is the result of combing prior art elements. Tang and Cho share the same field of endeavor of spatial-temporal feature extraction from frames. The motivation for the proposed modification would have been to have an image enhancement method comprising determining of the temporal-spatial feature representations comprises causing a size of the spatial feature representations to equal a size of the temporal feature representations; and determining the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations in order to use a SELFY block to perform the method as discussed to improve the video representation ability of a video-processing artificial neural network to perform the processing more effectively, see Cho, Par. [0069].
Regarding claim 10, Tang in view of Kadav, in combination, explicitly teaches the method of claim 1.
However, Tang in view of Kadav, in combination, does not explicitly teach wherein the temporal feature representations are feature maps generated by a first neural network, wherein the spatial-temporal features are feature maps generated by second neural network, and wherein a spatial-temporal feature map comprises spatial-temporal features of a corresponding image frame relative to other of the image frames.
In the same field of spatial-temporal feature extraction (Title and Abstract, Cho), Cho explicitly teaches wherein the temporal feature representations are feature maps generated by a first neural network (Par. [0086] discloses “in the case of a transformation from an STSS tensor…extract motion information throughout different temporal offset” indicating a STSS tensor transformer [first neural network] used to extract temporal features), wherein the spatial-temporal features are feature maps generated by second neural network (Par. [0067] discloses “CNN…to generate an STSS feature map” using the CNN to generate the spatial-temporal features [analogous to the second neural network as claimed, being the CNN]), and wherein a spatial-temporal feature map comprises spatial-temporal features of a corresponding image frame relative to other of the image frames (Par. [0070] discloses “an STSS model may define each position or local region of an image (or feature map) in terms of similarities with neighbors in time and space” indicating a spatial-temporal feature map comprises the features of corresponding image frames; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to have a process of determining spatial-temporal feature representations by generating temporal feature representations using a first neural network and generate the spatial-temporal feature representations using a second neural network. Thus in order to use a SELFY block to perform the method as discussed to improve the video representation ability of a video-processing artificial neural network to perform the processing more effectively, see Cho, Par. [0069]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement method comprising: extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set; generating inter-image temporal feature representations based on a local similarity between the spatial feature representations; determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations. Moreover, Tang’s determining of the temporal-spatial feature representations can be modified to be have the temporal feature representations are feature maps generated by a first neural network, wherein the spatial-temporal features are feature maps generated by second neural network, and wherein a spatial-temporal feature map comprises spatial-temporal features of a corresponding image frame relative to other of the image frames as taught in Cho.
Such a modification is the result of combing prior art elements. Tang and Cho share the same field of endeavor of spatial-temporal feature extraction from frames. The motivation for the proposed modification would have been to have a The image enhancement method of claim 1, wherein the generating of the temporal feature representations comprises: selecting a target spatial feature representation from among the spatial feature representations; comparing window regions of the target spatial feature representation with spatially-corresponding search regions of the spatial feature representations in order to use a SELFY, wherein the temporal feature representations are feature maps generated by a first neural network, wherein the spatial-temporal features are feature maps generated by second neural network, and wherein a spatial-temporal feature map comprises spatial-temporal features of a corresponding image frame relative to other of the image frames block to perform the method as discussed to improve the video representation ability of a video-processing artificial neural network to perform the processing more effectively, see Cho, Par. [0069].
Regarding claim 15, Tang in view of Kadav, in combination, explicitly teaches the electronic device of claim 11, wherein Tang explicitly teaches wherein, in order to determine the temporal-spatial feature representations, the instructions may be further configured to cause the one or more processors to (Par. [0060] discloses “multi-frame fusion may be implemented” moreover, Par. [0061] discloses “an inter-frame temporal relationship and an intra-frame spatial relationship are vitally important for multi-frame fusion…of aligned feature data” indicating a fusing of the spatial feature representations with the temporal feature representation, and the resulted fused feature data can be understood to be analogous to the recited temporal-spatial feature representations, Par. [0061] discloses “by means of the temporal and spatial attention mechanism based multi-frame fusion”).
However, Tang in view of Kadav, in combination, does not explicitly teach cause a size of the spatial feature representations to equal a size of the temporal feature representations; and determine the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations.
In the same field of spatial-temporal feature extraction (Title and Abstract, Cho), Cho explicitly teaches cause a size of the spatial feature representations to equal a size of the temporal feature representations (Par. [0109] discloses “the STSS feature map may have the same size as that of the video feature map”; moreover, Par. [0106] discloses “a spatial self-similarity map and a plurality of spatial cross-similarity maps for each position in a video feature map” indicating the video feature map include spatial feature representations; and Par. STSS feature map [spatial-temporal self-similarity feature map] includes temporal feature representations, therefore, it can be understood that the size of the spatial feature representations is equal to a size of the temporal feature representations, as claimed); and determine the temporal-spatial feature representations based on an elementwise addition operation (Par. [0109] discloses “the STSS feature map may have the same size as that of the video feature map…the electronic device may add the STSS feature map and the video feature map through an elementwise addition”; moreover, Par. [0102] discloses “the calculated final STSS feature map Z may be added, elementwise, to the video feature map V, and thus the SELFY block may operate as residual block for motion learning” indicating a result of offset-applied STSS feature map [analogous to the temporal-spatial feature representations as claimed being determined]) of feature values of the spatial feature representations and feature values of the temporal feature representations (Par. [0109] discloses “the STSS feature map may have the same size as that of the video feature map…the electronic device may add the STSS feature map and the video feature map through an elementwise addition”; moreover, Par. [0102] discloses “the calculated final STSS feature map Z may be added, elementwise, to the video feature map V, and thus the SELFY block may operate as residual block for motion learning” indicating the feature values of the spatial feature representations and feature values of the temporal feature representations; Therefore, it would have been obvious to one or ordinary skill of the art at the time the invention was made to have a process of determining of the temporal-spatial feature representations can be modified to be based on causing a size of the spatial feature representations to equal a size of the temporal feature representations; and determining the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations. Thus in order to use a SELFY block to perform the method as discussed to improve the video representation ability of a video-processing artificial neural network to perform the processing more effectively, see Cho, Par. [0069]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing data of the claimed invention was made to combine the teachings of Tang of an image enhancement device comprising extracting, by a spatial feature extractor, intra-image spatial feature representations from respective image frames of a burst image set; generating inter-image temporal feature representations based on a local similarity between the spatial feature representations; determining temporal-spatial feature representations of the respective image frames by fusing the spatial feature representations with the temporal feature representations. Moreover, Tang’s determining of the temporal-spatial feature representations can be modified to be based on causing a size of the spatial feature representations to equal a size of the temporal feature representations; and determining the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations as taught in Cho.
Such a modification is the result of combing prior art elements. Tang and Cho share the same field of endeavor of spatial-temporal feature extraction from frames. The motivation for the proposed modification would have been to have an image enhancement device comprising determining of the temporal-spatial feature representations comprises causing a size of the spatial feature representations to equal a size of the temporal feature representations; and determining the temporal-spatial feature representations based on an elementwise addition operation of feature values of the spatial feature representations and feature values of the temporal feature representations in order to use a SELFY block to perform the method as discussed to improve the video representation ability of a video-processing artificial neural network to perform the processing more effectively, see Cho, Par. [0069].
Pertinent Prior Art(s)
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Pace, Charles P. et. al. (“US 2006/0177140 A1”) discloses an apparatus and methods for processing video data are described. The invention provides a representation of video data that can be used to assess agreement between the data and a fitting model for a particular parameterization of the data. This allows the comparison of different parameterization techniques and the selection of the optimum one for continued video processing of the particular data. The representation can be utilized in intermediate form as part of a larger process or as a feedback mechanism for processing video data. When utilized in its intermediate form, the invention can be used in processes for storage, enhancement, refinement, feature extraction, compression, coding, and transmission of video data. The invention serves to extract salient information in a robust and efficient manner while addressing the problems typically associated with video data sources.
Wang, Xiaogang et. al., (“US 10825187 B2”) discloses a method includes: extracting, from the video, a 3-dimension (3D) feature block containing the target object; decomposing the extracted 3D feature block into a 2-dimension (2D) spatial feature map containing spatial information of the target object and a 2D spatial-temporal feature map containing spatial-temporal information of the target object; estimating, in the 2D spatial feature map, a location of the target object; determining, in the 2D spatial-temporal feature map, a speed and an acceleration of the target object; calibrating the estimated location of the target object according to the determined speed and acceleration; and tracking the target object in the video according to the calibrated location.
Xiaojie Jin et. al., (“US 20240395061 A1”) discloses a video processing method, apparatus, device, storage medium, and program product. The method includes: acquiring video data; obtaining, based on the video data, a temporal image feature with temporal information; determining, based on the temporal image feature, a target text feature in a set of text features that matches the temporal image feature; and obtaining, based on the target text feature, target text data corresponding to the video data.
Chaudhury; Subhajit et. al. (“US 12217191 B2”) discloses learning multimodal feature matching. The method includes training an image encoder to obtain encoded images. The method further includes training a common classifier on the encoded images by using labeled images. The method also includes training a text encoder while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings. The text encoder is further trained to match a distance of predicted text embeddings which is encoded by the text encoder to a fitted Gaussian distribution on the encoded images.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHUONG HAU CAI whose telephone number is (571)272-9424. The examiner can normally be reached M-F 8:30 am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHUONG HAU CAI/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights