Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 1/11/2024 was reviewed and the listed references were noted. The information disclosure statement (“IDS”) filed on 7/26/2025 does not include the mandatory IDS Size Fee Assertion; and thus, the references have not been considered.
Drawings
The 9-page drawings have been considered and placed on record in the file.
Status of Claims
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 6, 8, 9, 11, 12, 16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 12,230,030 B2) with EFD of May 10, 2022, in view of Gray et al. (US 2010/0309225 A1).
Regarding claim 1, Lee teaches, “A system comprising: at least one processor; at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising” (Lee, Column 14, Lines 61-62 discloses; “The server computing system 130 includes one or more processors 132 and a memory 134”), “identifying a visual marker in an image frame from a camera feed of a computing device” (Lee, Column 6, Lines 43-48 discloses; “Processing the image data and/or the scene data can include optical character recognition, object detection and recognition, pixel by pixel analysis, feature extraction then processing, image classification, object classification, object class determination, image segmentation, and/or environment or scene classification” and Lee, Column 10, Lines 9-13 discloses; “The voice input alongside the camera input can provide a conversational assistant that is visually aware of an environment, which can enable a user to be informed of an environment as they navigate through the environment.” Examiner interprets “identifying a visual marker” to be included within “optical character recognition, object detection and recognition, pixel by pixel analysis”). Lee also teaches, “identifying key points from the image frame comprising the visual marker”, (Lee, Column 6, Lines 43-48 discloses; “Processing the image data and/or the scene data can include optical character recognition, object detection and recognition, pixel by pixel analysis, feature extraction then processing, image classification, object classification, object class determination, image segmentation, and/or environment or scene classification” Examiner interprets “identifying key points” to be included within “pixel by pixel analysis” and “feature extraction” as disclosed by Lee). Additionally, Lee teaches, “filtering the set of matched images using a neural network trained to analyze the image frame and set of matched images”, (Lee, Column 14, Lines 21-26 discloses; “In some implementations, the user computing device 102 can implement multiple parallel instances of a single machine-learned model 120 (e.g., to perform parallel object recognition and tag generation across multiple instances of object recognition and filtering)” Examiner interprets the object recognition and tag generation to be synonymous with analyzing the image frame and the recited filtering). Furthermore, Lee teaches, “detecting the visual marker in an image of the filtered set of matched images”, (Lee, Column 14, Lines 27-29 discloses; “More particularly, the machine-learned model (e.g., a tag generation model) can process image data to recognize a plurality of objects in the scene depicted in the image data” Examiner interprets “visual marker” to be included in object detection). Moreover, Lee teaches, “in response to detecting the visual marker, causing presentation of a notification on a display unit of the computing device”, (Lee, Column 5, Lines 25-29 discloses; “The tags can be provided to the user via a user interface. The user may select a particular tag, which can cause the user interface to provide an augmented-reality experience that includes object-specific information overlaid over particular objects in the captured image”), “receiving a selection of the notification; and in response to receiving the selection of the notification, causing presentation of multi- media content associated with the visual marker on the display unit of the computing device” (Lee, Column 12, Lines 31-34 discloses; “Selection of a particular object and/or a tag associated with a particular object can be received and additional information on the particular object can be obtained and displayed”). Lee does not explicitly teach “comparing the image frame with a database of visual marker images using the identified key points; based on the comparison, identifying a set of matched images in the database of visual marker images”. Since Lee does not explicitly disclose these limitations, Examiner relies on the teachings of Gray, in an analogous field of endeavor. Specifically, Gray discloses, “comparing the image frame with a database of visual marker images using the identified key points”, (Gray Pg. 1, Para. 9 discloses; “For example, images obtained from a live video feed can be matched with a database of images to identify objects in the live video feed”), “based on the comparison, identifying a set of matched images in the database of visual marker images” (Gray Pg. 3, Para. 28 discloses; “Once the plurality of candidate images is selected, each image in the plurality of candidate images can be compared to the query image to identify matching images”).
Lee and Gray are both considered to be analogous to the claimed invention because they are in the same field of endeavor of detecting objects/features from a query image and matching that with other images or information. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Lee to incorporate the teachings of Gray in order to integrate a database containing images instead of conducting an internet search for images. One of ordinary skill in the art would have been motivated to combine the previously described system of Lee with the teachings of Gray to efficiently search for matching images by having a database with visual markers. Accordingly, it would have been obvious to combine Lee and Gray to obtain the above specified limitations.
Regarding claim 2, the combination of Lee in view of Gray teaches, “The system of claim 1, wherein the identified key points represent distinctive features of the image frame”, (Gray, Pg. 3, Para. 29 discloses; “At 502, keypoints (e.g., SURF keypoints) from features of the query image are compared to keypoints from features of the plurality of candidate images”). The proposed combination as well as the motivation for combining the Lee and Gray references presented in the rejection of claim 1, apply to claim 2 and are incorporated herein by reference. Thus, the system recited in claim 2 is met by Lee and Gray.
Regarding claim 6, the combination of Lee in view of Gray teaches, “The system of claim 1, wherein the database of visual markers is associated with a device location and device properties of the computing device”, (Gray, Pg. 1, Para. 13 discloses; “The geographical coordinate receiver 214 can acquire geographical coordinates (e.g., latitude and longitude) for the present location of the mobile internet device 102. In an example, the geographical coordinate receiver 214 is a global positioning system (GPS) receiver. In some examples, the mobile internet device 102 can also include other sensors such as one or more accelerometer to acquire acceleration force readings for the mobile internet device 102, one or more gyroscopes to acquire rotational force readings for the mobile internet device 102, or other sensors”). The proposed combination as well as the motivation for combining the Lee and Gray references presented in the rejection of claim 1, apply to claim 6 and are incorporated herein by reference. Thus, the system recited in claim 6 is met by Lee and Gray.
Regarding claim 8, the combination of Lee in view of Gray teaches, “The system of claim 1, wherein the visual marker is a free-form object”, (Lee, Abstract discloses; “Systems and methods for providing scene understanding can include obtaining a plurality of images, stitching images associated with the scene, detecting objects in the scene, and providing information associated with the objects in the scene” Examiner interprets a “free-form object” to be included in “detecting objects”).
Regarding claim 9, the combination of Lee in view of Gray teaches, “The system of claim 1, wherein the multi-media content is an augmented reality experience displayed on the display unit of the computing device”, (Lee, Abstract discloses; “The information may be provided in an augmented-reality experience via text or other user-interface elements anchored to objects in the images”).
Claims 11, 12, 16, and 18 recite methods with steps corresponding to the elements of the systems recited in Claims 1, 2, 6, and 8, respectively. Therefore, the recited steps of these claims are mapped to the proposed combination in the same manner as the corresponding elements in their corresponding system claims. Additionally, the rationale and motivation to combine the Lee and Gray references, presented in rejection of Claim 1, apply to these claims.
Claims 19 and 20 recite computer-readable storage media storing a program with instructions corresponding to the elements of the systems recited in Claim 1 and Claim 2, respectively. Therefore, the recited programming instructions of these claims are mapped to the proposed combination in the same manner as the corresponding elements in their corresponding system claims. Additionally, the rationale and motivation to combine the Lee and Gray references, presented in rejection of Claim 1, apply to these claims. Finally, the combination of Lee and Gray references discloses a computer readable storage medium (for example, see Lee, Column 13, Lines 64-67 discloses; “The memory 114 can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof).
Claims 3, 4, 13, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, in view of Gray, in further view of Holzschneider et al. (AU 2015220463 B2).
Regarding claim 3, the combination of Lee and Gray teaches, “The system of claim 1, further comprising: for each image in the database of visual marker images: identifying key points of the visual marker image”, (Gray, Pg. 3, Para. 26 discloses; “Once the images are downloaded, features are extracted from the images (e.g., with SURF or SIFT) and the extracted features are stored in the image database 410 for matching with the query image” Examiner interprets “key points” to fall within “features”). The combination of Lee and Gray does not explicitly teach “extracting a descriptor for each of the key points in the visual marker image; and generating a search index data structure, the search index data structure comprising the extracted descriptors”. Since the combination of Lee and Gray does not explicitly disclose these limitations, Examiner relies on the teachings of Holzschneider, in an analogous field of endeavor. Specifically, Holzschneider teaches, “extracting a descriptor for each of the key points in the visual marker image”, (Holzschneider, Pg. 12, Para. 51 discloses; “As will be described in more detail below, Image Feature Index database 143 provides a searchable index for feature descriptors extracted from reference images provided previously to the annotations database” Examiner interprets that the system of Holzschneider has already extracted the descriptors as mentioned). Additionally, Holzschneider teaches, “generating a search index data structure, the search index data structure comprising the extracted descriptors”, (Holzschneider, Pg. 12, Para. 51 discloses; “As will be described in more detail below, image Feature index database 143 provides a searchable index for feature descriptors extracted from reference images provided previously to the annotations database”).
The combination of Lee and Gray in view of Holzschneider are considered to be analogous to the claimed invention because they are all in the same field of detecting objects/features from a query image and matching that with other images or information. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Lee and Gray to incorporate the teachings of Holzschneider in order to create a searchable index database with images that already have feature descriptors for later matching. One of ordinary skill in the art would have been motivated to combine the previously described system of the combination of Lee and Gray with the teachings of Holzschneider to decrease the time and increase the accuracy of the system’s search and match ability. Accordingly, it would have been obvious to combine the combination of Lee in view of Gray and Holzschneider to obtain the invention in claim 3.
Regarding claim 4, the combination of Lee in view of Gray in further view of Holzschneider teaches, “The system of claim 3, wherein comparing the image frame with the database of visual marker images further comprises: extracting a query descriptor from each of the identified key points from the image frame” (Holzschneider, Pg. 4, Para. 10c discloses; “According to a first aspect the present invention provides a device for identifying and displaying additional information relating to an item of interest, the device comprising a processor for executing one or more modules comprising: an image acquisition module for acquiring a first query image including the item of interest; a feature descriptor generation module for extracting a plurality of first query image feature descriptors associated with the first query image”), “and searching for a match corresponding to the query descriptors within the search index data structure” (Holzschneider, Pg. 4, Para. 10c discloses; “and transmitting the first query fingerprint to a server, wherein the first query fingerprint enables the server to perform, based on the first query fingerprint, a similarity search in a database storing a plurality of reference image feature descriptors”). The proposed combination as well as the motivation for combining the Lee, Gray, and Holzschneider references presented in the rejection of claim 3, apply to claim 4 and are incorporated herein by reference. Thus, the system recited in claim 4 is met by Lee, Gray, and Holzschneider.
Claims 13 and 14 recite a method with steps corresponding to the elements of the system recited in Claims 3 and 4, respectively. Therefore, the recited steps of these claim are mapped to the proposed combination in the same manner as the corresponding elements in their corresponding system claims. Additionally, the rationale and motivation to combine the Lee, Gray, and Holzschneider references, presented in rejection of Claim 3, apply to these claims.
Claims 5, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, in view of Gray, in further view of Holzschneider, and still in further view of Ramnath et al. (US 2020/0143238 A1).
Regarding claim 5, the combination of Lee, Gray, and Holzschneider does not explicitly teach, “The system of claim 4, further comprising: storing the query descriptors in the search index data structure”. Since the combination of Lee, Gray, and Holzschneider does not explicitly disclose these limitations, Examiner relies on the teachings of Ramnath, in an analogous field of endeavor. Specifically, Ramnath teaches, “The system of claim 4, further comprising: storing the query descriptors in the search index data structure” (Ramnath, Pg. 5, Para. 30 discloses; “The stored local-feature descriptors may be stored on a local-feature database (e.g., located locally at the client device that captured the image, located at a remote server) that may, for example, include an index associating a plurality of local-feature descriptors with a plurality of AR targets”).
The combination of Lee and Gray, in view of Holzschneider, and still in view of Ramnath are considered to be analogous to the claimed invention because they are all in the same field of detecting objects/features from a query image and matching that with other images or information. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Lee, Gray, and Holzschneider to incorporate the teachings of Ramnath in order to locally store the query descriptors in an index for later use. One of ordinary skill in the art would have been motivated to combine the previously described system of the combination of Lee, Gray, and Holzschneider with the teachings of Ramnath to allow a user to access their query history, and/or add the query descriptors to the already existing database of descriptors. Accordingly, it would have been obvious to combine the combination of Lee in view of Gray, in further view of Holzschneider with Ramnath to obtain the invention in claim 5.
Claim 15 recites a method with steps corresponding to the elements of the system recited in Claim 5. Therefore, the recited steps of this claim are mapped to the proposed combination in the same manner as the corresponding elements in its corresponding system claim. Additionally, the rationale and motivation to combine the Lee, Gray, Holzschneider, and Ramnath references, presented in rejection of Claim 5, apply to this claim.
Claims 7, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lee, in view of Gray, in further view of Ramnath et al. (US 2020/0143238 A1).
Regarding claim 7, the combination of Lee and Gray, does not teach, “The system of claim 1, wherein detecting the visual marker in the image of the filtered set of matched images further comprises: analyzing each image in the filtered set of matched images using a geometric verification algorithm”. Since the combination of Lee and Gray does not explicitly disclose these limitations, Examiner relies on the teachings of Ramnath, in an analogous field of endeavor. Specifically, Ramnath teaches, “The system of claim 1, wherein detecting the visual marker in the image of the filtered set of matched images further comprises: analyzing each image in the filtered set of matched images using a geometric verification algorithm”, (Ramnath, Pg. 5, Para. 30 discloses; “In particular embodiments, the computing device may determine a match between two local-feature descriptors by identifying a suitable transformation that may cause an extracted local-feature descriptor to match a stored local-feature descriptor using a verifier process (e.g., a geometric verifier process)”). Furthermore, the combination of Lee, Gray, and Ramnath teaches, “based on the analysis, detecting the visual marker in a verified image of the filtered set of matched images”, (Lee, Column 14, Lines 27-29 discloses; “More particularly, the machine-learned model (e.g., a tag generation model) can process image data to recognize a plurality of objects in the scene depicted in the image data” Examiner interprets “visual marker” to be included in object detection).
The combination of Lee and Gray in view of Ramnath are considered to be analogous to the claimed invention because they are all in the same field of detecting objects/features from a query image and matching that with other images or information. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Lee and Gray to incorporate the teachings of Ramnath in order to efficiently filter the set of matched images with a geometric verification algorithm. One of ordinary skill in the art would have been motivated to combine the previously described system of the combination of Lee and Gray with the teachings of Ramnath to have a point of verification to ensure the matched images contain the visual marker. Accordingly, it would have been obvious to combine the combination of Lee in view of Gray with Ramnath to obtain the invention in claim 7.
Regarding claim 10, the combination of Lee, Gray, and Ramnath teaches, “The system of claim 1, wherein the multi-media content is selectable by a user of the computing device”, (Ramnath, Pg. 11, Para. 60 discloses; “In particular embodiments, a concept node 804 may represent a third-party webpage or resource hosted by a third-party system 770. The third-party webpage or resource may include, among other elements, content, a selectable or other icon, or other inter-actable object (which may be implemented, for example, in JavaScript, AJAX, or PHP codes) representing an action or activity. As an example and not by way of limitation, a third-party webpage may include a selectable icon such as “like,” “check-in,” “eat,” “recommend,” or another suitable action or activity.”). The proposed combination as well as the motivation for combining the Lee, Gray, and Ramnath references presented in the rejection of claim 7, apply to claim 10 and are incorporated herein by reference. Thus, the system recited in claim 10 is met by Lee, Gray and Ramnath.
Claim 17 recites a method with steps corresponding to the elements of the system recited in Claim 7. Therefore, the recited steps of this claim are mapped to the proposed combination in the same manner as the corresponding elements in its corresponding system claim. Additionally, the rationale and motivation to combine the Lee, Gray, and Ramnath references, presented in rejection of Claim 7, apply to this claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUSTIN M. OAKES whose telephone number is (571)272-9379. The examiner can normally be reached 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUSTIN M OAKES/Examiner, Art Unit 2662
/Siamak Harandi/Primary Examiner, Art Unit 2662