DETAILED ACTION
This action is in response to the application filed on August 14, 2024. Claims 1-21 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of US Provisional Application No. 63/614,113, filed December 22, 2023. Claims 1-21 have been afforded the benefit of this filing date.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 11-12, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Micusik et al, US 20230267691 in view of Appalaraju et al, US 10467526 in view of Peng et al, US 20250157055.
Regarding claim 1, Micusik teaches
A computer-implemented method comprising (see Micusik, Paragraph [0001], “the present disclosure addresses systems and methods for detecting changes in a scene using with a visual tracking system”):
accessing an initial dataset of real images of an environment, wherein for an image of the initial dataset there exists at least one matching image in the initial dataset and a set of non-matching images, the matching image represents similar scene of the environment as in the image of the initial dataset (see Micusik, Paragraph [0030], “accessing a first set of images and corresponding pose data in a first coordinate system associated with a first user session of an augmented reality (AR) device, accessing a second set of images and corresponding pose data in a second coordinate system associated with a second user session of the AR device, aligning the first coordinate system with the second coordinate system based mapping the pose data of the first set of images to the pose data of the second set of images, identifying the first set of images corresponding to a second image from the second set of images based on the pose data of the first set of images being determined spatially closest to the pose data of the second image after aligning the first coordinate system and the second coordinate system,”);
identifying a plurality of domain shifts, wherein each domain shift of the plurality of domain shifts corresponds to a potential change in the environment (see Micusik, Paragraph [0030], “detecting changes in a scene,” and Paragraph [0003], “The optical sensor in the AR/VR device can also be used to detect changes in a scene (e.g., a physical environment) captured by the same AR/VR device during two distinct sessions,” detecting changes in a scene (e.g. physical environment) is considered to be a plurality of domain shifts);
generating, for each image of the initial dataset and for each domain shift of the plurality of domain shifts, a synthetic image (see Micusik, Paragraph [0030], “generating, using a trained neural network, a synthesized image from the first set of images, determining differences between features of the second image from features of the synthesized image, and identifying changes based on the differences.”);
identifying a plurality of matching pairs in the initial dataset, wherein each matching pair of the plurality of matching pairs comprises of a first image and a second image and both images captures similar aspects or scene of the environment (see Micusik, Paragraph [0030], “identifying the first set of images corresponding to a second image from the second set of images based on the pose data of the first set of images being determined spatially closest to the pose data of the second image after aligning the first coordinate system and the second coordinate system”);
Micusik does not expressively teach
determining, for each matching pair of the plurality of matching pairs in the initial dataset, a plurality of tentative pairs, wherein each tentative pair of the plurality of tentative pairs includes at least one synthetic image, the tentative pairs are generated by making image pairs of: the first image with the synthetic images of the second image, and the synthetic images of the first image with the second image;
However, Appalaraju in a similar invention in the same field of endeavor teaches
determining, for each matching pair of the plurality of matching pairs in the initial dataset, a plurality of tentative pairs, wherein each tentative pair of the plurality of tentative pairs includes at least one synthetic image, the tentative pairs are generated by making image pairs of: the first image with the synthetic images of the second image, and the synthetic images of the first image with the second image (see Appalaraju, Col 8, Lines 1-9, “in order to generate a positive pair of images, a slight modification may be applied to the first image of a pair, and the modified or synthesized image may be used as the second image of the pair—that is, instead of using images that were generated by some external entity, a new similar image may be created at the image analytics service to obtain a positive image pair”);
The combination of Micusik and Appalaraju are analogous art because they are both in the same field of endeavor of determining changes/similarities between images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to generate a positive pair of images and apply a modification to the first image of a pair and the modified or synthesized image be used as the second image of the pair as taught in the system of Appalaraju in the system of Micusik to help obtain a sufficient number of positive image pairs to train the model (Appalaraju, Col 6, Lines 1-4).
Micusik in view of Appalaraju does not expressively teach
determining, for each tentative pair of the plurality of tentative pairs, geometric correspondences;
determining, for each tentative pair of the plurality of tentative pairs, whether the tentative pair is a valid pair or not based on a degree of the geometric correspondences between images of the tentative pair inside an area of interest;
generating an extended dataset by combining the initial dataset with the valid pairs;
and training an image retrieval model using a contrastive loss function with weights assigned to each matching pair and corresponding valid pairs in the extended dataset.
However, Peng in a similar invention in the same field of endeavor teaches
determining, for each tentative pair of the plurality of tentative pairs, geometric correspondences (see Peng, Paragraph [0067], “the electronic device may identify various regions based on consistent geometry (e.g., geometric consistency) and instance consistency (e.g., identical pose information) of the frame pair. For example, based on the geometric consistency and the instance consistency of the frame pair, the electronic device may identify a background region, a rigid dynamic object region, a non-rigid dynamic object region, an occluded region, and an out-of-boundary region”);
determining, for each tentative pair of the plurality of tentative pairs, whether the tentative pair is a valid pair or not based on a degree of the geometric correspondences between images of the tentative pair inside an area of interest (se Peng, Paragraph [0076], “the electronic device may obtain the target motion field by updating (e.g., optimizing or initializing) the motion field based on a matching level (e.g., a distance or a similarity level),” matching level is considered to be determining whether the tentative pair is a valid pair);
generating an extended dataset by combining the initial dataset with the valid pairs (see Peng, Paragraph [0076], “Based on the fused embedding features, the electronic device may soft-group pixels into geometrically consistent regions (e.g., pixels corresponding to the same or similar object) in the frame pair. For reference, fused embedding features of point pairs (e.g., a pixel point pair or pixel pair) with geometric consistency may be similar to each other. Therefore, the electronic device may obtain the target motion field by updating (e.g., optimizing or initializing) the motion field based on a matching level (e.g., a distance or a similarity level) between fused embedding features of pixel point pairs”);
and training an image retrieval model using a contrastive loss function with weights assigned to each matching pair and corresponding valid pairs in the extended dataset (see Peng, Paragraph [0153], “the electronic device may combine the second training loss (e.g., a mask loss) corresponding to each sample frame pair and the first training loss (e.g., a scene flow estimation loss) of each sample frame pair to obtain a combined training loss of the AI network. The electronic device may adjust parameters (e.g., weights) of the AI network based on the combined training loss”).
The combination of Micusik, Appalaraju, and Peng are analogous art because they are all in the same field of endeavor of determining changes/similarities between images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to determine a matching level (distance or similarity level); group pixels into geometrically consistent regions and updating the motion field by based on a matching level; obtain a combined training loss the of AI network; and determine 3D geometric consistency of the object in the frame pair as taught in the apparatus of Peng in the system of Micusik in view of Appalaraju to improve the matching quality of a pixel pair in an end-to-end AI network (Peng, Paragraph [0190]).
Regarding claim 4, Micusik in view of Appalaraju in view of Peng further teaches the computer-implemented method of claim 1,
wherein the area of interest is determined by computing a matching function which returns a set of matches that are geometrically consistent between two images of a matching pair, the area of interest is composed of image pixels corresponding to the set of matches (Peng, Paragraph [0190], “the electronic device may consider accurate matching or correspondence between densely populated pixels, such as, 3D geometric consistency of an object in the frame pair and an instance (or “object instance” herein) consistency (e.g., since a rigid motion embedding feature is not supervised in the training phase, it may be easy to disrupt the 3D geometric consistency and the instance consistency of the object in the frame pair, which may reduce the pixel matching quality of an occluded region). The 3D geometric consistency and the instance consistency of the object in the frame pair may be helpful to improve the matching quality of a pixel pair in an end-to-end AI network. The electronic device may use a new AI network (e.g., the AI network 210) based on RAFT-3D”).
The rationale of claim 1 has been applied herein.
As per claim 11, Claim 11 claims a system comprising: one or more data processors; and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including: to complete the same limitations as Claim 1. Therefore, the rejection and rationale are analogous to that made in Claim 1.
Micusik further teaches A system comprising: one or more data processors (see Micusik, Paragraph [0079], “As shown in FIG. 9, high-speed circuitry 918 includes high-speed processor 920”);
and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform actions including (see Micusik, Paragraph [0078], “The head-wearable apparatus 902 includes a memory 922 which stores instructions to perform a subset or all of the functions described herein memory 922 can also include storage device”).
As per claim 12, Claim 12 claims the same limitations as Claim 4 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale is analogous to that made in Claim 4.
As per claim 18, Claim 18 claims a computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including: to complete the same limitations as Claim 1. Therefore, the rejection and rationale are analogous to that made in Claim 1.
Micusik further teaches A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including (see Paragraph [0079], “As shown in FIG. 9, high-speed circuitry 918 includes high-speed processor 920” and Micusik, Paragraph [0078], “The head-wearable apparatus 902 includes a memory 922 which stores instructions to perform a subset or all of the functions described herein memory 922 can also include storage device,” a memory which stores instructions to perform functions that includes storage device is a computer-program product tangibly embodied in a non-transitory machine-readable storage medium).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Micusik et al, US 20230267691 in view of Appalaraju et al, US 10467526 in view of Peng et al, US 20250157055 in view of Lee et al, KR 20230108997A.
Regarding claim 2, Micusik in view of Appalaraju in view of Peng does not expressively teach the computer-implemented method of claim 1, further comprising:
receiving a query image;
and predicting a position and/or orientation of a camera device used to capture the query image by processing the query image using the image retrieval model.
However, Lee in a similar invention in the same field of endeavor teaches
receiving a query image (see Lee, Paragraph [0034], “Specifically, the control server (100) can use the received surrounding image as a query image”);
and predicting a position and/or orientation of a camera device used to capture the query image by processing the query image using the image retrieval model (see Lee, Paragraph [0034], “Specifically, the control server (100) can use the received surrounding image as a query image and search for a reference image corresponding to the query image within the map information. Here, since the position and pose of the measuring equipment corresponding to the reference image may be set, it is possible to obtain the current position and pose of the robot (1) by comparing the reference image with the surrounding image. That is, the position and pose of the robot (1) corresponding to the query image can be obtained through local feature matching”).
The combination of Micusik, Appalaraju, Peng, and Lee are analogous art because they are all in the same field of endeavor of determining changes/similarities between images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, to receive surrounding image as a query image; obtain the current position and pose of the robot by comparing the reference image with the surrounding image as taught in the method of Lee in the system of Micusik in view of Appalaraju in view of Peng possible to perform more accurate visual positioning (Lee, Paragraph [0015]).
Claim(s) 8, 16, and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Micusik et al, US 20230267691 in view of Appalaraju et al, US 10467526 in view of Peng et al, US 20250157055 in view of Zhang et al, CN 115861701 A.
Regarding claim 8, Micusik in view of Appalaraju in view of Peng does not expressively teach the computer-implemented method of claim 1, further comprising:
wherein the image retrieval model is trained using an AdamW optimizer for longer time with a cosine learning rate schedule.
However, Zhang in a similar invention in the same field of endeavor teaches
wherein the image retrieval model is trained using an AdamW optimizer for longer time with a cosine learning rate schedule (see Zhang, Paragraph [n0051], “Figure 2 is a flowchart of the remote sensing image retrieval stage of the present invention,” and Paragraph [n0057], “Since the dataset contains tens of thousands of images, it cannot be loaded into memory all at once. It needs to be divided into small batches using a data loader, and the small batches of data are then fed into the neural network. AdamW was chosen as the optimizer, with a batch size of 280, a basic learning rate of 0.001, and cosine decay”).
The combination of Micusik, Appalaraju, Peng, and Zhang are analogous art because they are all in the same field of endeavor of determining changes/similarities between images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, for remote sensing image retrieval to use AdamW as the optimizer and a learning rate with cosine decay as taught in the method of Zhang in the system of Micusik in view of Appalaraju in view of Peng to automatically retrieve desired images based on visual content similarity to the query image. (Zhang, Paragraph [n0002]).
As per claim 16, Claim 16 claims the same limitations as Claim 8 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale is analogous to that made in Claim 8.
As per claim 21, Claim 21 claims the same limitations as Claim 8 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale is analogous to that made in Claim 8.
Allowable Subject Matter
Claim(s) 3, 5-7, 9-10, 13-15, 17, and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DOMINIQUE JAMES whose telephone number is (703)756-1655. The examiner can normally be reached 9:00 am - 6:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571)270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DOMINIQUE JAMES/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights