DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Application’s amendment/response filed on 01/12/2026, which has been entered and made of record. No Claims has been cancelled.
Claims 1-20 are pending in the application.
The objection to claim 16 has been withdrawn.
Response to Arguments
Applicant’s arguments, see Page 2 in the remarks, filed on 01/12/2026, with respect to the rejection(s) of claims 1 and 20 under 35 USC § 103 regarding the usage of reference Pati et al. (US 20230169666 A1) as have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of new prior arts of Shree et al. (US 20220284609A1) and Zhang et al. (US 20220156943A1).
Applicant’s arguments with respect to claims 1 and 20 regarding the newly-added “to generate first segmentation masks. . .” and “by determining matches between the first segmentation masks . . .” limitations are fully considered but are moot in view of the new grounds of rejection represented this Office Action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 8, 10, 13, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Shree et al. (US 20220284609A1) (Hereinafter referred to as Shree) in view of Zhang et al. (US 20220156943A1) (Hereinafter referred to as Zhang) and in further view of Pati et al. (US 20230169666 A1) (Hereinafter referred to as Pati) and in further view of Wang et al. (US 20180158199 A1) (Hereinafter referred to as Wang).
Regarding Claim 1, Shree discloses A method of processing an image, the method comprising: (See Abstract, “Techniques are described for identifying correspondences between images. . .”)
segmenting both a first image and a second image to generate first feature representations of the first image and second feature representations of the second image, and generating feature representation pairs, (See [0094], “For example, referring to FIG. 7A, a first feature representation 720A is selected from representations 712 of the first image (in FIG. 7A). A second feature representation 720B is selected from representations 712 or 714 of the second image (in FIG. 7B). . . As discussed below, in one or more embodiments, multiple pairings of feature representations from a pair of images may be made and a homography generated for each pairing.” Here, Shree teaches a first and second image, selecting a first and second feature in each respective image, and then being able to generate multiple pairings of feature representations.
Further see [0060], “a feature representation comprises a geometry of a boundary of the feature, such as coordinates for a point, vertices for a line, or vertices and/or edges of polygon shape.” Also see Figs. 7A-7B showing feature representations.
Further see [0157], “In one or more embodiments, determining the structure portion and the one or more feature portions comprises segmenting or categorizing the pixels of the digital image into a plurality of predicted components.” In this case, Shree teaches to “segmenting both a first image and a second image” in order to get the feature representations.)
by determining matches between the first feature representations and the second feature representations, each feature representation pair having a first feature representation of the first image and a second feature representation of the second image; (See [0094], “For example, referring to FIG. 7A, a first feature representation 720A is selected from representations 712 of the first image (in FIG. 7A). A second feature representation 720B is selected from representations 712 or 714 of the second image (in FIG. 7B). . . As discussed below, in one or more embodiments, multiple pairings of feature representations from a pair of images may be made and a homography generated for each pairing. For example, each feature representation in the first digital image may be paired with one or more feature representations from the second digital image and the resulting feature pairs may be analyzed in serial or in parallel.”
Also see [0105] In one or more embodiments, correctly matching feature representations, i.e., feature representations that correspond to the same feature, . . .”)
generating local homography matrices of the first image with respect to the second image, based on the feature representation pairs, and indirectly based on the first image, and the second image; and (See [0094], “As discussed below, in one or more embodiments, multiple pairings of feature representations from a pair of images may be made and a homography generated for each pairing.” In this case, these generated homography matrices can be considered as “local homography matrices”, and it would be based on the feature representation pairs. Since the feature representation pairs are generated based on the first and second image, that means that the homography matrices are indirectly based on the first and second image.)
aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the feature representation pairs, the first image, and the second image. (See [0083], “The homography is generated to align a representation of a first feature disposed on the surface of the first image with a representation of a second feature in the second image.”
Also see [0094], “As discussed below, in one or more embodiments, multiple pairings of feature representations from a pair of images may be made and a homography generated for each pairing.” In this case, aligning the features of the first and second image can be considered as “aligning the first image with the second image”, and this would be based on the local homography matrices, and the feature representation pairs, and indirectly based on the first and second image, as those are used to generate the feature representation pairs.)
However, Shree fails to explicitly disclose segmenting both a first image and a second image to generate first segmentation masks of the first image and second segmentation masks of the second image, and generating segmentation mask pairs
by determining matches between the first segmentation masks and the second segmentation masks, each segmentation mask pair having a first segmentation mask of the first image and a second segmentation mask of the second image;
generating local homography matrices of the first image with respect to the second image, based on the segmentation mask pairs, the first image, and the second image; and
generating a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image.
Zhang teaches using segmentation masks for features. (See Abstract, “For instance, a system can determine a first segmentation feature associated with a first segmentation mask of a first image frame. The system can determine a second segmentation feature associated with a second segmentation mask of a second image frame.”
Thus, in combination with Shree which teaches having multiple feature representations from a first and second image and multiple pairs, the feature representations would have segmentation masks to represent them.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shree with Zhang to include using segmentation masks to represent feature representations.
The motivation to combine Shree with Zhang would have been obvious as both Shree and Zhang are within the same field of art of processing features of a first and second image (See Zhang Abstract). Shree already describes feature representations as being a boundary outlining a feature within the image (See Shree [0060]), and a segmentation masks is a similar idea of isolating that specific feature from the rest of the image, and thus the combination would be obvious.
However, Shree in view of Zhang still fails to explicitly disclose generating local homography matrices of the first image with respect to the second image, based on the segmentation mask pairs, the first image, and the second image; and
generating a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image.
Pati teaches generating local homography matrices of the first image with respect to the second image, based on the segmentation mask pairs, the first image, and the second image; (See [0061], “The AI model is herein referred to as the TMPN and may . . . a Homography net, a CNN, or another suitable model.”
See [0042], “where the first image 244, the first mask 246, the second image 245, and the second mask 247 are input into neural network module 240 to generate a transformation matrix (e.g., including shift vectors) based thereon.” In this case, Pati teaches using a first and second image along with the masks to generate a transformation (homography) matrix. Even though Pati does not teach and dissuades having multiple segmentation masks, Shree in view of Zhang already teaches the idea of using of multiple segmentation masks pairs in generate multiple (local) homography matrices, and Pati is simply used to teach the idea of using the first and second image along with the masks to generate a homography matrix.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shree in view of Zhang with Pati to include using first and second images in the process of generating a homography matrix.
The motivation to combine Shree in view of Zhang with Pati would have been obvious as both Shree and Pati are within the same field of generating homography matrices between two images (See Pati [0042] and [0061]). In particular, Pati teaches using a neural network model that takes the images and masks as input (See Pati [0042]) to output a transformation matrix (in the case of Homography Net, a homography matrix). The benefit of using images and masks together when generating a homography matrix is that it can improves the accuracy and robustness of the homography matrices and would thus result in improved image alignment.
However, Shree in view of Zhang and Pati still fail to explicitly disclose generating a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image.
Wang teaches generating a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image. (See [0097], “For instance, the act 1250 can include warping the subsequent image 106 based on the homography of each local region of the plurality of local regions to generate a new aligned image 116 that is pixel-wise aligned to the reference image 104.” Here, Wang directly teachings aligning based on the local homography matrices.)
Note that Wang additionally teaches generating local homography matrices of the first image with respect to the second image. (See [0064], “As mentioned above briefly, determining the homography of the reference image 104 to the subsequent image 106 is described in greater detail below in regard to FIGS. 5 and 6. . . may be performed within a step for determining a homography for each local region 105 of the plurality of local regions 105.” In this case, reference image and subsequent image correspond to first and second image and determining a homography for each local region of the plurality of local regions correspond to generating local homography matrices.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shree in view of Zhang and Pati with Wang to include generating a synthetic image by aligning based on the local homography matrices.
The motivation to combine would have Shree in view of Zhang and Pati with Wang been obvious as Wang is an art within the same field of aligning images and generating homography matrices (See Wang Abstract). The benefit of having local homography matrices and using them for aligning images is that it result in higher quality aligned image. See Wang [0006], “subdivide a reference image and a subsequent image of the burst images into a plurality of local regions . . . Based on the matching feature points, the systems and methods determine a homography (i.e., a motion transformation) that enables the systems and methods to warp the subsequent image and generate a higher quality new aligned image that is pixel-wise aligned to the reference image.”
Regarding Claim 2, Shree in view of Zhang, Pati, and Wang disclose The method of claim 1, wherein the first and second image are images of a scene comprised of regions respectively corresponding to the segmentation mask pairs, and wherein each segmentation mask pair’s images both correspond to the segmentation mask pair’s region. (See Shree [0094] and Zhang Abstract teaching segmentation mask pairs.
See Wang [0093], “For example, act 1210 can include subdividing each of a reference image 104 and a subsequent image 106 into a plurality of local regions.”
Further See Wang [0095], “In particular, the act 1230 may include determining matching pairs 112 of feature points 108 between the reference image 104 and the subsequent image 106.” Lastly see Wang Fig. 12 showing determining feature points of regions and matching points between the images. All of this implies that the each segmentation mask pair’s images both correspond to the segmentation mask pair’s region. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 3, Shree in view of Zhang, Pati, and Wang disclose The method of claim 1, wherein the segmenting the first image and the second image comprises generating first initial segmentation masks of the first image and second initial segmentation masks of the second image; and (See Shree [0094], “For example, referring to FIG. 7A, a first feature representation 720A is selected from representations 712 of the first image (in FIG. 7A). A second feature representation 720B is selected from representations 712 or 714 of the second image (in FIG. 7B). . .” See Zhang Abstract teaching segmentation mask for features.)
wherein generating the segmentation mask pairs comprises post-processing the first initial segmentation masks and the second initial segmentation masks. (See Shree [0094], “For example, each feature representation in the first digital image may be paired with one or more feature representations from the second digital image and the resulting feature pairs may be analyzed in serial or in parallel.” One can consider the feature matching and pairing the feature representations to be “post-processing” the first and second feature representation in order to generate the segmentation mask pairs. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 8, Shree in view of Zhang, Pati, and Wang disclose The method of claim 1, wherein the generating of the local homography matrices of the first image with respect to the second image comprises generating the local homography matrices by applying a first neural network to the segmentation mask pairs, the first image, and the second image. (See Pati [0061] teaching Homography net (a first neural network). Further see Pati, [0042], “where the first image 244, the first mask 246, the second image 245, and the second mask 247 are input into neural network module 240 to generate a transformation matrix (e.g., including shift vectors) based thereon.”
Also see Wang [0064] teaching local homography for each local region. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 10, Shree in view of Zhang, Pati, and Wang disclose The method of claim 8, further comprising: segmenting both a first training image and a second training image of a training image pair and generating a training segmentation mask pairs of the training image pair; and (See Shree [0094] and Zhang Abstract teaching segmentation mask pairs.
See Pati [0019], “the deep learning model is trained with a plurality of training data sets, each including two training pairs and associated ground truth. Together, the training pairs may include an image pair comprised of a first image and a second image, . . . The training pairs may further comprise one or more masks, where a first mask is generated based on the first image and/or a second mask is generated based on the second.”)
generating the first neural network by training with the training image pair and the training segmentation mask pairs, based on the training segmentation mask pairs of the training image pair, (See Pati [0010], “FIG. 4 shows flow chart of an example method for training the TMPN to generate a transformation matrix based on input data;” Also see Pati Fig. 4 showing training of the TPMN, which can be a Homography net (first neural network) which includes training with the training image pair and the training segmentation mask pairs.)
wherein the training segmentation mask pair comprises a segmentation mask for one region of the first training image and a segmentation mask for a region corresponding to the one region of the first training image in the second training image. (See Wang [0093] teaching regions for the images. See Wang [0095] and Fig. 12 teaching feature points of regions and matching points between the images. All of this implies that the masks can be matched to have corresponding regions. Then Pati teaches that they can be used for training purposes. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 13, Shree in view of Zhang, Pati, and Wang disclose The method of claim 1, wherein a segmentation mask pair are formed by determining that a segmentation mask of the first image matches a segmentation mask of the second image. (See Shree [0094] and Zhang Abstract teaching segmentation mask pairs.
Also see Shree [0094], “For example, each feature representation in the first digital image may be paired with one or more feature representations from the second digital image and the resulting feature pairs may be analyzed in serial or in parallel.”
Also see Shree [0105], “In one or more embodiments, correctly matching feature representations, i.e., feature representations that correspond to the same feature, . . .” The motivation to combine would have been similar to Claim 1 rejection motivation.)
Regarding Claim 18, Shree in view of Zhang, Pati, and Wang disclose The method of claim 1, wherein each homography matrix aligns a region of the first image with a corresponding region of the second image. (See Shree [0083], “The homography is generated to align a representation of a first feature disposed on the surface of the first image with a representation of a second feature in the second image.”
Also see Shree Figs. 7A-7B showing feature representations. In this case, the feature representations shown can be considered as “regions” of the first and second image as they are the specific features of those images in which the homography matrices are used to align them.)
Regarding Claim 20, Shree in view of Zhang, Pati, and Wang disclose An electronic device comprising: one or more processors; and memory storing computer-executable instructions configured to cause the one or more processors to: (See Shree [0303], “According to one embodiment, the techniques described herein are implemented by at least one computing device. . . may include at least one general purpose hardware processor programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage”)
segmenting both a first image and a second image to generate first segmentation masks of the first image and second segmentation masks of the second image, and generating segmentation mask pairs, by determining matches between the first segmentation masks and the second segmentation masks, each segmentation mask pair having a first segmentation mask of the first image and a second segmentation mask of the second image;
generating local homography matrices of the first image with respect to the second image, based on the segmentation mask pairs, the first image, and the second image; and generating a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image. (The above limitations are similar to that of Claim 1 and is therefore rejected under a similar rationale as Claim 1.)
Allowable Subject Matter
Claims 4-7, 9, 11-12, 14-17, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 4, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: selecting N first segmentation masks from the first segmentation masks and selecting N second segmentation masks from the second segmentation masks; merging an unselected first segmentation mask into one of the N first segmentation masks and merging an unselected second segmentation mask into one of the N second segmentation masks; and generating the segmentation mask pairs by performing mask matching between the N first segmentation masks and the N second segmentation masks, wherein each of the first segmentation masks and the second segmentation masks is a segmentation mask of a connected region, and wherein an area of each of the N first segmentation masks and the N second segmentation masks is greater than a first threshold value. Thus Claim 4 contains allowable subject matter.
Claims 5-7 depend upon Claim 4 and thus also contains allowable subject matter.
Regarding Claim 9, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: generating an encoding pyramid feature for the first image, based on a feature map of the first image and first segmentation masks in the segmentation mask pairs; generating an encoding pyramid feature for the second image, based on a feature map of the second image and second segmentation masks in the segmentation mask pairs; and predicting the local homography matrices based on the encoding pyramid feature for the first image and the encoding pyramid feature for the second image. Thus Claim 4 contains allowable subject matter.
Regarding Claim 11, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: segmenting the first training image and the second training image respectively from one training image pair and generating initial segmentation masks of the first training image and initial segmentation masks of the second training image; and generating training segmentation mask pairs of the training image pair by post-processing the initial segmentation masks of the first training image and the initial segmentation masks of the second training image. Thus Claim 11 contains allowable subject matter.
Claim 12 depends upon Claim 11 and thus also contains allowable subject matter.
Regarding Claim 14, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: generating distorted images by applying the respective local homography matrices to the first or second image; and applying weights to the distorted images and fusing the weighted distorted images. Thus Claim 14 contains allowable subject matter.
Claims 15-17 depend upon the base of Claim 14 and thus also contains allowable subject matter.
Regarding Claim 19, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: merging two initial segmentation masks determined to have a same classification, the classifications of the initial segmentation masks determined from the first or second image. Thus Claim 19 contains allowable subject matter.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611