DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC§ 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 11 (and 12 by dependence) are rejected under 35 U.S.C. 112(b) as being indefinite. The phrase "the at least one processing device is to:” omits "configured," which appears in the parallel apparatus limitations, rendering the intended functional recitation unclear. Applicant should amend to "the at least one processing device is configured to:."
Claims 13 and 20 are rejected under 35 U.S.C. 112(b) as being indefinite. In each, the
trailing clause "at least one of the tile-based registration or the feature-based registration not
performed or performed differently in other segments in the image frames associated with other portions of the scene" omits a verb/connective (e.g., "is not performed or is performed
differently"), so the metes and bounds of the differential-registration limitation are unclear.
Applicant should amend to mirror the corresponding language of method claim 6 (" ... is
performed in the one or more segments ... and is not performed or is performed differently in
other segments ... ").
Claim 14 is rejected under 35 U.S.C. 112(b) as being indefinite. Claim 14 depends from
apparatus claim 8 but recites a method step in gerund form ("further comprising: blending the
reference frame and the one or more aligned non-reference frames to generate a final image of
the scene") that is not tied to the "at least one processing device" being configured to perform it. Reciting a method step within an apparatus claim renders it unclear whether infringement occurs when the device is made or when the blending is performed. IPXL Holdings v. Amazon.com, Inc., 430 F.3d 1377, 1384, 77 USPQ2d 1140, 1145 (Fed. Cir. 2005).
Applicant should amend to "wherein the at least one processing device is further configured to blend ... " ( cf. method claim 7).
Claims 11-14 and 20 are examined below on the merits as best understood.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 7-12, and 14-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhen et al., US 2021/0042941 A1 (hereinafter “Zhen”) in view of Le et al., (US 10,911,691 B1 (hereinafter “Le”) in view of Braun et al., US 2012/0162449 A1 (hereinafter “Braun”).
Claim 1.
Zhen and Le disclose a method comprising: obtaining, using at least one processing device of an electronic device, multiple image frames capturing a scene (ZHEN: "A method includes receiving a reference image and a non-reference image" (Abstract); the image frames are captured for multi-frame fusion (¶ 28). This teaches obtaining multiple image frames of a scene.);
selecting, using the at least one processing device, a reference frame among the image frames (ZHEN: "one image frame can be used as a reference image frame 255, and the other image frame can be used as a non-reference image frame 260," and "the shorter exposure image frame may be selected as the reference image frame 255" (¶ 57).);
and aligning, using the at least one processing device, each of one or more non-reference frames among the image frames with the reference frame by: performing tile-based registration of the non-reference frame to the reference frame (ZHEN: "A tile-based motion vector image breaks up the image into multiple tiles and tries to find motion map like a flow ... The result is a two-dimensional (2D) map of motion vectors of the reference frame" (¶ 60).);
(ZHEN: the image alignment operation "can use global Oriented FAST and Rotated BRIEF (ORB) features and local features from a block search to align the image frames" (¶ 49); explicit feature-based registration is supplied by Le below.);
(ZHEN: the structure-guided refinement combines the tile-based local motion with global and similarity constraints in a single weighted optimization to produce the warp, where "The feature points 515 in the reference image frame 505 are centers of the tiles on the finest scale" (¶ 87); the aggregation of an independent feature-based motion field is addressed below.); and
warping the non-reference frame based on the aggregated motion vectors to generate an aligned non-reference frame (ZHEN: "use motion vectors from the motion vector estimation 240 to compute a global geometric transformation, which could be described by an affine matrix" (¶ 82).).
Zhen discloses tile-based registration, use of feature information, and warping, but does not expressly describe a separate feature-based registration that generates a distinct second set of motion vectors. However, Le, in the same field of multi-frame image registration, teaches
feature-based registration of image frames to a reference frame (LE: the device "performs
keypoint detection and descriptor extraction on the luminance components. Based on the
obtained keypoints, the electronic device 101 estimates a homography to warp other image
frames to a selected base image frame" (C10:L30-37).).
Braun further teaches that, for a given frame pair, both a tile-based motion field and a feature-point motion field are computed (BRAUN: a "tile-vector based Hierarchical Block- Matching search algorithm for deriving motion vectors of the selected feature points" (Abstract); a full block-matching search per tile produces a tile motion vector (¶¶ 18-19), and a motion vector is computed for each feature point (¶¶ 11-12).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to perform both the tile-based registration of Zhen and the feature-based registration of Le on the same reference/non-reference frame pair, and to aggregate the resulting first (tile-based) and second (feature-based) motion vectors to warp the non-reference frame. This is a combination of prior art motion-estimation techniques according to known methods (MPEP 2143(A)) to yield the predictable result of a single alignment warp, with a reasonable expectation of success because Zhen already fuses tile-based motion with additional constraints in a single warp optimization and Braun already computes both motion fields for the same frames. The reason to aggregate the two fields is Zhen's own recognition that they are complementary: tile-based motion vectors "are not reliable on flat areas due to an absence of features" (Zhen, ¶ 63), whereas feature-based (keypoint/ORB) registration performs well in textured regions. Aggregating the two complementary motion fields therefore yields a more robust alignment across both flat and textured scene content than either field alone.
Claim 8.
Zhen and Le disclose an electronic device comprising: at least one imaging sensor configured to
capture multiple image frames of a scene (ZHEN: image frames captured by an electronic
device 101 having sensors (FIG. 1; ¶ 57).); and at least one processing device configured to:
obtain the image frames; select a reference frame among the image frames; and align each of one or more non-reference frames with the reference frame; wherein, to align each non-reference frame, the at least one processing device is configured to: perform tile-based registration (ZHEN: tile-based motion vector map (¶ 60).); (LE: keypoint detection and descriptor extraction, homography estimation (C10:L30-37).);
(ZHEN: compute a global geometric
transformation/affine matrix from the motion vectors (¶ 82).). The struck limitations are
supplied and combined for the reasons given for claim 1.
Claim 15.
Zhen and Le disclose a non-transitory machine readable medium containing instructions that
when executed cause at least one processor to: obtain multiple image frames; select a reference frame; and align each non-reference frame, wherein the instructions to align comprise instructions to: perform tile-based registration (ZHEN: ¶ 60.); p
(LE: C10:L30-37.);
and warp the non-reference frame based on the aggregated motion vectors (ZHEN: ¶ 82.).
Zhen discloses storage of instructions executed by a processor (¶ 30; FIG. 1). The struck
limitations are supplied and combined for the reasons given for claim 1.
Claims 2, 9, and 16.
Zhen, Le, and Braun discloses that performing the tile-based registration comprises dividing the non-reference frame into tiles, comparing each tile to a neighborhood of tiles in the reference frame, selecting a tile based on the comparison, and generating at least one of the first motion vectors based on the selected tile (ZHEN: "split the reference image frame into multiple tiles 310 ... search for a corresponding tile in a neighborhood of a non-reference image frame 330 for each tile in the reference image frame" (¶ 76), producing "a motion vector for each of the patches into which the reference frame 255 is divided" (¶ 61).).
Claims 3, 10, and 17.
Zhen, Le, and Braun disclose that performing the feature-based registration comprises extracting features from the non-reference frame, comparing each feature to a corresponding feature in the reference frame, selecting one or more features, and generating at least one of the second motion vectors based on the selected features (LE: "performs keypoint detection and descriptor extraction ... Based on the obtained keypoints, the electronic device 101 estimates a homography" (C10:L30-37). BRAUN: computes a "motion vector of the feature point" by comparing macroblocks about the feature point between frames (¶¶ 11-12).).
Claims 4, 11 , and 18.
Zhen, Le, and Braun discloses that warping the non-reference frame based on the aggregated motion vectors comprises determining a warping based on the motion vectors and applying the warping to generate the aligned non-reference frame (ZHEN: "compute a global geometric transformation, which could be described by an affine matrix" and warp the non-reference
frame accordingly (¶ 82).). (Claim 11 is examined as best understood in view of the§ 112(b)
rejection above.)
Claims 5, 12, and 19.
Zhen, Le, and Braun disclose using a weighted perspective model to generate a transformation matrix to be applied to the non-reference frame (LE: "estimates a homography to warp other image frames" (C10:L30-37), a homography being a perspective transformation matrix. ZHEN:
applies a weighted refinement optimization combining "a local alignment term," "a global
constraint term," and "a similarity term" (Claim 7).).
It would have been obvious to generate the perspective (homography) transformation matrix of Le using a weighting of the more reliable motion vectors as taught by Zhen, i.e., a weighted perspective model, as a predictable combination of known techniques that improves alignment accuracy by de-emphasizing unreliable motion vectors.
Claims 7 and 14.
Zhen, Le, and Braun discloses blending the reference frame and the one or more aligned non-reference frames to generate a final image of the scene (ZHEN: "The aligned image frame is then blended in a blending operation" 220 to combine the frames (¶ 51).). (Claim 14 is examined as best understood in view of the § 112(b) rejection above.)
Allowable Subject Matter
Claim 6 is objected to as being dependent upon a rejected base claim, but would be
allowable if rewritten in independent form including all of the limitations of the base claim and
any intervening claims. Claims 13 and 20 recite subject matter that would likewise be allowable
over the prior art of record, but stand rejected under 35 U.S.C. 112(b) above; each would be
allowable if rewritten in independent form and the indefiniteness is corrected
Conclusion
The following prior art made of record, but not relied upon, is considered pertinent to the Applicant’s disclosure. CN 108694705 A (Zhejiang University) discloses a multi-frame image registration and fusion denoising method that selects a reference frame among multiple frames and registers each remaining frame to the reference frame using both block-matching motion estimation and feature-based (e.g. SIFT) motion-vector estimation, followed by pixel-domain and transform-domain fusion of the registered images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
Prosecution Insights