DETAILED ACTION
This Action is in response to Applicant’s response filed on 01/21/2026. Claims 1-20 are still pending in the present application. This Action is made FINAL.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/21/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
Claim Objection: The amended claims filed on 01/21/2026 overcomes the Claim Objection in the previous office action.
Response to Arguments
Applicant's arguments filed on 01/21/2026 have been fully considered but they are not persuasive. In the present application, applicant argues that “Claim 1 recites one or more features that are not taught or suggested by Luo and Pope, individually or in combination. For example, no combination of Luo and Pope teaches or suggests: "determining the additional content [to be displayed as an AR overlay] based on the hyperlinks that are included in the particular additional content, which is determined based on the particular pair determined when the particular GTIF product-token matches the particular pair," as claimed. Neither reference teaches/suggests determining the additional content [to be displayed as an AR overlay] based on hyperlinks; much less determining the additional content based on the hyperlinks that are included in the particular additional content, which is determined based on the particular pair determined when the particular GTIF product-token matches the particular pair, as claimed.
Since neither Luo nor Pope teaches or suggests "determining the additional content [to be displayed as an AR overlay] based on the hyperlinks that are included in the particular additional content, which is determined based on the particular pair determined when the particular GTIF product-token matches the particular pair," as claimed, no combination of Luo and Pope can possibly teach/suggest Claim 1. Hence, Claim 1 is patentable over Luo and Pope, individually or in combination. (Remark Page 3)
In response to Argument, the Examiner respectfully disagree. Luo discloses in Fig. 10 and “an interface 1050 is provided showing additional information based at least in part on the product metadata described above, which may be received and displayed by the augmented reality content generator module 706 on the client device 102.As shown, production information 1060 is included (e.g., product description, price, manufacturer, vendor, selection graphical item to purchase the product). As further shown, media content 1065 (e.g., an embedded video), and additional media content 1070 are included in the interface 1050. In this example, the media content 1065 can be played directly within the interface 1050, whereas the additional media content 1070 may include respective descriptions and links to associated media content (e.g., respective video).” is consider as “wherein the particular additional content comprises hyperlinks to the additional content”. Also, Lou discloses in Fig. 16 and in interface 1600 is provided showing information of product based at least in part on product metadata, which may be received and displayed by the augmented reality content generator module 706 on the client device 102 … the media content 1620 can be played directly within the interface 1600, whereas the additional media content 1630 may include respective descriptions and links to associated media content (e.g., respective video). Additional information 1622 related to the product, or related to the media content (e.g., a description of the media content 1620) can be included in the interface 1050.)” is read as “based on the hyperlinks, determining the additional content”. AND “an interface 1650 (which may be displayed by the augmented reality content generator module 706 on the client device 102) is shown with augmented reality content 1660 indicating the product that has been applied as augmented reality content 1670 on the captured image data corresponding to a representation of the user's face.” is interpreted as “causing, by the client application, an augmented reality (AR) overlay to be rendered on the user device, device” (Paragraphs 177-178)
Furthermore, Fig.18 and information 1810 (e.g., a type of media content in the form of text) is included showing information based on a detected facial characteristic of the user (e.g., skin or face condition, eye condition, and the like), which can be based on the analysis performed by the product identification module 704 … A selectable graphical item 1812 (e.g., a button) is included to enable navigation to an external source (e.g., external web site) to view additional information, … media content 1830 includes a respective preview or graphical representation (e.g., image or short video) of associated media content (e.g., respective videos of tutorial or beauty routines, web links, etc.) which make demonstrate how to apply the product shown in the graphical item 1820 is interpreted as “wherein the particular additional content comprises hyperlinks to the additional content; based on the hyperlinks, determining the additional content; and causing, by the client application, an augmented reality (AR) overlay to be rendered on the user device, device”. (Paragraphs 184-187)
Additionally, Luo discloses The product catalog system 124 can store the received information into the database 120. Based at least on information retrieved from the database 120 related to the product, the client device 102 can provide for display (e.g., rendering on a UI of the messaging client application 104) an AR experience for the product.” (Paragraph 110) is interpreted as “determining particular additional content based on the particular pair, and causing, by the client application, an augmented reality (AR) overlay to be rendered on the user device”. Also, Luo discloses in Fig.7 and the augmented reality content generator module 706 utilizes different object or facial detection processes to detect objects or a face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and render an AR experience based on the selected AR content generator from the carousel interface for display on a given client device (e.g., the client device 102). … The augmented reality content generator module 706 performs rendering of content for display by the messaging client application 104 based on data provided by at least one of the aforementioned modules. For example, the augmented reality content generator module 706 performs various operations based on algorithms or techniques that correspond to animations or providing visual or auditory effects, based on the product information, to the received image data … the augmented reality content generator module 706 utilizes a graphical processing pipeline to perform graphical operations to render the message for display. The augmented reality content generator module 706 implements, in an example, an extensible rendering engine which supports multiple image processing operations corresponding to respective media overlays.” (Paragraphs 122-124). Furthermore, Figs. 13-15 and an interface 1550 is provided, which may displayed by the augmented reality content generator module 706 on the client device 102 ... In this example, the position of the selectable graphical item 1560 corresponds to a “medium” setting (as the slider is in the middle), which results in augmented reality content 1564 to be rendered, based on this setting, on a portion of the user's face in the captured image data. As also shown, a selectable graphical item 1570 is provided, which when selected through user input can cause another interface to be displayed on the client device, such as a rendering of the user's face with the applied makeup in the form of the augmented reality content applied to the captured image data.” (Paragraph 175) is interpreted as “wherein the AR overlay is generated by compositing the additional content onto a live or static view of the physical object, using the spatial coordinates of the detected features as anchor points for the overlay, such that the additional content is visually registered to the physical object in the digital image."
Therefore, Luo determines particular additional content based on the hyperlink (link, website service and/or Web link) that included in the additional media content and a specific pair of data.
The Examiner states that in light of MPEP 2111, the Examiner has interpreted the claims properly. Specifically, during patent prosecution, the pending claims must be “given their broadest reasonable interpretation assistant with the specification.” The Examiner has interpreted the claim language in reference to the specification. Because applicant has the opportunity to amend the claims during prosecution, given a claim in its broadest reasonable interpretation will reduce the possibility that the claim, once issued will be interpreted more or broadly than is justified. Although the cited reference is different from the invention disclosed, the language of Applicant's claims is sufficiently broad to reasonably read on the cited reference. A broad reading does not constitute “teaching away.”
Further, it has been held that nonpreferred embodiments failing to assert discovery beyond that known in the art does not constitute a “teaching away” unless such disclosure criticizes, discredits, or otherwise discourages the solution claimed. In re Susi, 440 F.2d 442, 169 USPQ 423 (CCPA 1971), In re Gurley, 27 F.3d 551, 554, 31 USPQ2d 1130, 1132 (Fed. Cir. 1994), In re Fulton, 391 F.3d 1195, 1201, 73 USPQ2d 1141, 1146 (Fed. Cir. 2004), (see MPEP §2124).
Disclosed examples and preferred embodiments do not constitute a teaching away from a broader disclosure or nonpreferred embodiments. In re Susi, 440 F.2d 442, 169 USPQ 423 (CCPA 1971). “A known or obvious composition does not become patentable simply because it has been described as somewhat inferior to some other product for the same use.” In re Gurley, 27 F.3d 551, 554, 31 USPQ2d 1130, 1132 (Fed. Cir. 1994) (The invention was directed to an epoxy impregnated fiber-reinforced printed circuit material. The applied prior art reference taught a printed circuit material similar to that of the claims but impregnated with polyester-imide resin instead of epoxy. The reference, however, disclosed that epoxy was known for this use, but that epoxy impregnated circuit boards have “relatively acceptable dimensional stability” and “some degree of flexibility,” but are inferior to circuit boards impregnated with polyester-imide resins. The court upheld the rejection concluding that applicant’s argument that the reference teaches away from using epoxy was insufficient to overcome the rejection since “Gurley asserted no discovery beyond what was known in the art.” 27 F.3d at 554, 31 USPQ2d at 1132.). Furthermore, “[t]he prior art’s mere disclosure of more than one alternative does not constitute a teaching away from any of these alternatives because such disclosure does not criticize, discredit, or otherwise discourage the solution claimed….” In re Fulton, 391 F.3d 1195, 1201, 73 USPQ2d 1141, 1146 (Fed. Cir. 2004). (MPEP §2124).
Claim Status
Claim(s) 1, 3-8, 10-15 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (U.S. 20210312533 A1; Luo), in view of Pope, Arthur et al (“Learning object recognition models from images.”; Pope).
Claim(s) 2, 9 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (U.S. 20210312533 A1; Luo), in view of Pope, Arthur et al (“Learning object recognition models from images.”; Pope), and in further view of Collart (U.S. 20200104522 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-8, 10-15 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (U.S. 20210312533 A1; Luo), in view of Pope, Arthur et al (“Learning object recognition models from images.”; Pope).
Regarding claim 1, Luo discloses a method (Paragraph 198: “FIG. 20 is a flowchart illustrating a method 2000 to provide selected information and media content in response to scanning a product (e.g., beauty product)”) comprising:
using a client application executing on a user device, receiving a digital image of a physical object; (Fig. 7 and Paragraph 117: “ Paragraph 117: “The image data receiving module 702 receives images captured by a client device 102. For example, an image is a photograph captured by an optical sensor (e.g., camera) of the client device 102. An image includes one or more real-world features, such a user's face, or a physical object(s) detected in the image.”; Fig. 20 ; Paragraph 199: “At operation 2002, the image data receiving module 702 receives image data including a representation of a physical item. In an example, the physical item is a beauty product”)
generating, by the client application, a particular graph of transform-invariant features (GTIF) product-token associated with the digital image of the physical object; (Fig. 7; Paragraph 118-120: “The product identification module 704 utilizes different object detection processes to detect objects in the image, such as a physical item corresponding to a product that a user wants to extract product metadata from, or a physical indicator of identification (e.g., barcode) corresponding to the physical item. … the product identification module 704, in an example embodiment, utilizes a set of classifiers that classify an image received from a camera of a mobile computing device into one or more classes. In an embodiment, an example set of image classifier determines whether the image includes a physical identification indicator containing text, barcode pattern(s), or QR code pattern(s), and the like. … The product identification module 704 can perform operations (e.g., a process) for extracting product metadata from a recognized object corresponding to a physical identification indicator of the physical item in the image,”; Figs. 9-20 and Paragraph 134-145; Paragraph 202: “At operation 2006, the product identification module 704 extracts product metadata based on the determined object.”)
processing, by the client application, the digital image using a computer-vision pipeline, implementing feature detection and using a scale-invariant feature transform (SIFT), to extract a plurality of transform-invariant features, (Paragraph 120: “the product identification module 704 can perform operations (e.g., a process) for extracting product metadata from a recognized object corresponding to a physical identification indicator of the physical item in the image,”; Paragraph 160-167: “Fig.13 example interfaces for scanning a user's face and presenting various information based on detected features or facial characteristics, and the focal region 1330 can be analyzed by the product identification module 704 or other component of the client device 102) using techniques described below (read as “the digital image using a computer- vision pipeline”) to attempt to identity feature points corresponding to one or more facial features of the user's face and other facial characteristics … the aforementioned detection techniques can determine the facial features and facial characteristics using a number of different techniques … other object recognition processes can be used. … Moreover, techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.”) and construct a graph data structure representing spatial relationships among the transform-invariant features; (Fig.3 and Paragraph 55: “The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. … Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).” ; Fig.13 and Paragraph 168-169: “Fig.13, an interface 1350 is provided with information and graphical items as a result of the analysis performed … the interface 1350 includes a graphical item 1360 (e.g., floating window or dialog box), graphical item 1370, and graphical item 1380 that are superimposed over the captured image of the user's face. Each of the aforementioned graphical items includes information regarding a detected feature(s) or facial characteristic(s), such as a face shape (e.g., oval), indication of skin health, etc.”, The person of ordinary skill in the art would understand that the Lou teaches generating a GTIF product associated with the digital image of the physical object by using object recognition processes as scale-invariant feature transform (SIFT).)
based on, at least in part, the graph data structure, determining whether the particular GTIF product-token, generated by the computer-vision pipeline, associated with the digital image, matches a particular pair of a set of GTIF product-token pairs; (Paragraph 75: “the database 120 also stores data of products in a product table 316, which enables the product catalog system 124 to perform operations related to providing an augmented reality experience with respect to a product (e.g., a given physical item that may be available for purchase or sale). In an example, the product table 316 includes a directory (e.g., listing) of products and their associated product identifiers, which can be compared against product metadata provided by the product catalog system 124.”; Paragraph 108-109: “The product catalog system 124 can receive at least the product ID from the client device 102 (or the product catalog system 124 as discussed below), and perform a lookup, search, or select operation on the product table 316 to retrieve the product metadata from the database 120. … In an example, the product catalog system 124 can send a request message to a respective server for obtaining metadata related to a given physical item. The request message may include, for example, the product ID.” ; Fig.20 and Paragraph 202: “At operation 2006, the product identification module 704 extracts product metadata based on the determined object. In an embodiment, extracting the product metadata based on the determined object further includes comparing the identified object to a library of objects, each object from the library of objects including associated metadata with product information corresponding to a product, determining that the identified object matches a particular object from the library of object based at least in part on a confidence score being above a threshold value, and providing the associated metadata of the particular object.”)
wherein the set of GTIF product-token pairs comprises one or more pairs comprising known GTIF product-token and data determined for a user, a user device or a user event; (Paragraph 55-57: “The database 120 includes message data stored within a message table 314. The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of type, any entity regarding which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown). The entity graph 304 furthermore stores information regarding relationships and associations between entities. Such relationships may be social, professional (e.g., work at a common corporation or organization) interested-based or activity-based, … The database 120 also stores annotation data, in the example form of filters, in an annotation table 312. … Other types of filters include geolocation filters (also known as geo-filters) which may be presented to a sending user based on geographic location.”; Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”)
in response to determining that the particular GTIF product-token matches the particular pair, determining particular additional content based on the particular pairs, (Fig. 20 and Paragraph 203-204: “At operation 2008, the product identification module 704 sends, to a server, the product metadata to determine second product metadata associated with the product metadata. The second product metadata includes media content from an external source, the external source being different than a messaging platform utilized by the client device. At operation 2010, the product identification module 704 receives, from the server, the second product metadata, the second product metadata including additional information related to the physical item.” Paragraph 110: “In response to the request message, the product catalog service system 602 can provide a response message to the product catalog system 124, which includes, in an example embodiment, the information from the database 620 in response to the request. The product catalog system 124 can store the received information into the database 120. Based at least on information retrieved from the database 120 related to the product, the client device 102 can provide for display (e.g., rendering on a UI of the messaging client application 104) an AR experience for the product.”)
wherein the particular additional content comprise hyperlink to the additional content; based on the hyperlinks, determining the additional content; (Figs. 10, 15-16 and 18 ; Paragraph 142: “an interface 1050 is provided showing additional information based at least in part on the product metadata described above, which may be received and displayed by the augmented reality content generator module 706 on the client device 102.As shown, production information 1060 is included (e.g., product description, price, manufacturer, vendor, selection graphical item to purchase the product). As further shown, media content 1065 (e.g., an embedded video), and additional media content 1070 are included in the interface 1050. In this example, the media content 1065 can be played directly within the interface 1050, whereas the additional media content 1070 may include respective descriptions and links to associated media content (e.g., respective video).”; Paragraphs 184-187; “information 1810 (e.g., a type of media content in the form of text) is included showing information based on a detected facial characteristic of the user (e.g., skin or face condition, eye condition, and the like), which can be based on the analysis performed by the product identification module 704 … A selectable graphical item 1812 (e.g., a button) is included to enable navigation to an external source (e.g., external web site) to view additional information, … media content 1830 includes a respective preview or graphical representation (e.g., image or short video) of associated media content (e.g., respective videos of tutorial or beauty routines, web links, etc.) which make demonstrate how to apply the product shown in the graphical item 1820”) and
causing, by the client application, an augmented reality (AR) overlay to be rendered on the user device; (Paragraphs 122: “the augmented reality content generator module 706 utilizes different object or facial detection processes to detect objects or a face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and render an AR experience based on the selected AR content generator from the carousel interface for display on a given client device (e.g., the client device 102” ; Paragraph 178: “an interface 1650 (which may be displayed by the augmented reality content generator module 706 on the client device 102) is shown with augmented reality content 1660 indicating the product that has been applied as augmented reality content 1670 on the captured image data corresponding to a representation of the user's face.”)
wherein the AR overlay is generated by compositing the additional content onto a live or static view of the physical object, using the spatial coordinates of the detected features as anchor points for the overlay, such that the additional content is visually registered to the physical object in the digital image. (Fig.7 and Paragraphs 122-124: “the augmented reality content generator module 706 utilizes different object or facial detection processes to detect objects or a face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and render an AR experience based on the selected AR content generator from the carousel interface for display on a given client device (e.g., the client device 102). … The augmented reality content generator module 706 performs rendering of content for display by the messaging client application 104 based on data provided by at least one of the aforementioned modules. For example, the augmented reality content generator module 706 performs various operations based on algorithms or techniques that correspond to animations or providing visual or auditory effects, based on the product information, to the received image data … the augmented reality content generator module 706 utilizes a graphical processing pipeline to perform graphical operations to render the message for display. The augmented reality content generator module 706 implements, in an example, an extensible rendering engine which supports multiple image processing operations corresponding to respective media overlays.” ; Figs. 13-15 and Paragraph 175: “an interface 1550 is provided, which may displayed by the augmented reality content generator module 706 on the client device 102 ... In this example, the position of the selectable graphical item 1560 corresponds to a “medium” setting (as the slider is in the middle), which results in augmented reality content 1564 to be rendered, based on this setting, on a portion of the user's face in the captured image data. As also shown, a selectable graphical item 1570 is provided, which when selected through user input can cause another interface to be displayed on the client device, such as a rendering of the user's face with the applied makeup in the form of the augmented reality content applied to the captured image data.”)
However, Luo does not disclose construct a graph data structure representing spatial relationships among the transform-invariant features of the physical object, wherein the graph data structure has a plurality of linked nodes that represent how some of the transform-invariant features of the physical object are linked with other transform-invariant features identified for the physical object;
Pope discloses receiving a digital image of a physical object; generating, by the client application, a particular graph of transform-invariant features (GTIF) product-token associated with the digital image of the physical object; (3. Approach: “Any image input to the system, whether for training or for recognition, is first represented as an attributed graph by a perceptual organization process. This image graph contains explicit information about the presence and spatial arrangement of significant features in the image. A series of image graphs, each an example of some object's appearance, is combined to form a model graph by a model learning procedure.”)
processing, by the client application, implementing feature detection and using a scale-invariant feature transform (SIFT), to extract a plurality of transform-invariant features, construct a graph data structure representing spatial relationships among the transform-invariant features of the physical object, wherein the graph data structure has a plurality of linked nodes that represent how some of the transform-invariant features of the physical object are linked with other transform-invariant features identified for the physical object; (3.1: Image and model graphs: “Once detected in an image, a feature is represented by a token that records the type of feature plus its image location, orientation, and scale. … These attributes are always expressed in a manner invariant with respect to translation, rotation, and scaling … An image graph records the hierarchy of tokens created from a single image: each node represents a token and the directed arcs represent composition and abstraction. … a model is represented like an image: as a graph whose nodes are feature tokens and whose arcs represent composition and abstraction relations among features. So that it can represent not just one appearance but a distribution of them the model graph contains additional statistical information”; 4. Experimental result: “Figure 2 shows the system learning a model for a cup, and figure 3 shows it recognizing the partly-occluded cup and two other objects in an image. The cup model contains 138 tokens and the image yields 308. About 30000 token matches were attempted in one minute to find an optimal match, which involves 61 token pairs.)
based on, at least in part, the graph data structure, determining whether the particular GTIF product-token associated with the digital image, matches a particular pair of a set of GTIF product-token pairs; (3.6: Modeling learning Procedure: “ From the first training image’s graph we select the connected subgraph containing the largest-scale token; this becomes the initial model. Each of its tokens has one attribute vector and is credited with one successful match. The next image’s graph is then compared with the model graph to find the best match.”; 4. Experiment Results: “Models were created for the two bottles using six training images of each. (Table 2 summarizes this process for one bottle.) Each model was then matched with two subimages from figure 3: one containing the identical object, the other containing its counterpart.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Luo by a model graph by a model learning procedure that is taught by Pope, to make the invention that systems and methods for authorizing rendering of objects in three-dimensional spaces; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving recognize an object in an image as well as enhancing the feature identified in image processing technique.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 3, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token comprises one or more digital hyperlinks indicating one or more locations in a storage device at which the particular additional content and identifiers of additional products have been stored; (Paragraph 172: “A selectable graphical item 1412 (e.g., a button) is included to enable adding this information regarding the detected face shape to a message. Media content 1420 includes respective descriptions and previews or graphical representations (e.g., images or short videos) of associated media content (e.g., respective videos of tutorial or beauty routines, web links, etc.). …the additional media content 1430 includes an embedded video that can be played within the interface 1400, without requiring the user to navigate to a. different interface or screen. In an embodiment, any or all of the aforementioned media content is provided (and stored) by an external source (e.g., a. third party) separate from the client device 102 and the messaging server system 108.”) wherein the particular additional content and the additional products are related to the physical object, have been constructed around the physical object, or are associated with circumstance related to an event related to the physical object or the digital image. (Paragraph 142: “an interface 1050 is provided showing additional information based at least in part on the product metadata described above, which may be received and displayed by the augmented reality content generator module 706 on the client device 102. … the media content 1065 can be played directly within the interface 1050, whereas the additional media content 1070 may include respective descriptions and links to associated media content (e.g., respective video). Additional information 1067 related to the product, or related to the media content (e.g., a description of the media content 1065) can be included in the interface 1050.”)
Regarding claim 4, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token is a complex data structure that is generated using advanced computer-based techniques that include one or more: encoding spatial representations of certain features identified in the digital image or determining a set of invariant features that are specific to the digital image; wherein the set of invariant features includes features that remain invariant of any 2D transformation performed on the features of the digital image. (Paragraph 62-66 ; Paragraph 118: “ The product identification module 704 utilizes different object detection processes to detect objects in the image, such as a physical item corresponding to a product that a user wants to extract product metadata from, or a physical indicator of identification (e.g., barcode) corresponding to the physical item. … optical character recognition (OCR) can be used as a primary image analysis technique or to enhance other processes. Features (e.g., shape, size, color and text) of the image can be extracted. image processing processes may include sub-processes such as … segmentation, blob extraction, pattern recognition, barcode and data matrix code reading, gauging (measuring object dimensions), positioning, edge detection, color analysis, filtering (e.g. morphological filtering) and template matching (finding, matching, or counting specific patterns).”)
Regarding claim 5, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token pair, of the set of GTIF product-token pairs, represents one or more of: one or more of relationships between a plurality of transform-invariant features identified for the digital image, or one or more relationships between the plurality of transform-invariant features identified for the digital image and other transform-invariant features identified for other products. (Paragraph 107: “he product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, Universal Product Code (UPC) codes, QR codes … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”; Paragraphs 62-66 and 118)
Regarding claim 6, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the digital image has a plurality transform invariant features and a corresponding plurality of GTIF product-tokens; wherein a GTIF product-token is used to determine whether the GTIF product-token matches a particular pair of the set of GTIF product-token pairs; (Paragraph 75: “The database 120 also stores data of products in a product table 316, which enables the product catalog system 124 to perform operations related to providing an augmented reality experience with respect to a product (e.g., a given physical item that may be available for purchase or sale). In an example, the product table 316 includes a directory (e.g., listing) of products and their associated product identifiers, which can be compared against product metadata provided by the product catalog system 124”) wherein a GTIF product-token pair comprises additional context data that include one or more of: location data determined based on GPS location data obtained from one or more of: the location of the user device, a photo, an address of an event, or an address of customers or users; social relationship data of a creator or a recipient of the digital image; or time based data determined based on one or more of: a time of an event, a time when a photo was taken, or a time when a message was sent. (Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, Universal Product Code (UPC) codes, QR codes … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”;
Regarding claim 7, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the determining that the particular GTIF product- token matches the particular pair is a search that requires comparisons between non-directed graphs having a plurality of nodes, wherein the nodes represent transform invariant features, wherein a time for comparison performed as a series of instructions on computing machinery increases based on a number of comparisons, and wherein a number of transform invariant features exceeds practical limits of user interaction time; wherein a GTIF product-token is generated using one or more of: a scale-invariant feature transform feature recognition method (SIFT), a simultaneous localization and mapping feature recognition method (SLAM), or a speed up robust features feature recognition method (SURF). (Paragraph 108; Paragraph 167: “other object recognition processes can be used. … Other techniques include feature-based techniques, such as may utilize interpretation trees, geometric hashing, or invariance analysis. Moreover, techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.”)
Regarding claim 8, Luo discloses one or more non-transitory computer readable storage media (Fig.25: memory/storage 2506) storing one or more instructions (Fig.25: instruction 2510) which, when executed by one or more processors, (Fig.25: processors 2504) cause the one or more processors (Paragraph 240: “FIG. 25 shows a diagrammatic representation of the machine 2500 in the example form of a computer system, within which instructions 2510 (e.g., software, a program, an application, an apples, an app, or other executable code) for causing the machine 2500 to perform any one or more of the methodologies discussed herein may be executed.”) to perform:
using a client application executing on a user device, receiving a digital image of a physical object; (Fig. 7 and Paragraph 117: “ Paragraph 117: “The image data receiving module 702 receives images captured by a client device 102. For example, an image is a photograph captured by an optical sensor (e.g., camera) of the client device 102. An image includes one or more real-world features, such a user's face, or a physical object(s) detected in the image.”; Fig. 20 ; Paragraph 199: “At operation 2002, the image data receiving module 702 receives image data including a representation of a physical item. In an example, the physical item is a beauty product”)
generating, by the client application, a particular graph of transform-invariant features (GTIF) product-token associated with the digital image of the physical object; (Fig. 7; Paragraph 118-120: “The product identification module 704 utilizes different object detection processes to detect objects in the image, such as a physical item corresponding to a product that a user wants to extract product metadata from, or a physical indicator of identification (e.g., barcode) corresponding to the physical item. … the product identification module 704, in an example embodiment, utilizes a set of classifiers that classify an image received from a camera of a mobile computing device into one or more classes. In an embodiment, an example set of image classifier determines whether the image includes a physical identification indicator containing text, barcode pattern(s), or QR code pattern(s), and the like. … The product identification module 704 can perform operations (e.g., a process) for extracting product metadata from a recognized object corresponding to a physical identification indicator of the physical item in the image,”; Figs. 9-20 and Paragraph 134-145; Paragraph 202: “At operation 2006, the product identification module 704 extracts product metadata based on the determined object.”)
processing, by the client application, the digital image using a computer-vision pipeline, implementing feature detection and using a scale-invariant feature transform (SIFT), to extract a plurality of transform-invariant features, (Paragraph 120: “the product identification module 704 can perform operations (e.g., a process) for extracting product metadata from a recognized object corresponding to a physical identification indicator of the physical item in the image,”; Paragraph 160-167: “Fig.13 example interfaces for scanning a user's face and presenting various information based on detected features or facial characteristics, and the focal region 1330 can be analyzed by the product identification module 704 or other component of the client device 102) using techniques described below (read as “the digital image using a computer- vision pipeline”) to attempt to identity feature points corresponding to one or more facial features of the user's face and other facial characteristics … the aforementioned detection techniques can determine the facial features and facial characteristics using a number of different techniques … other object recognition processes can be used. … Moreover, techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.”) and construct a graph data structure representing spatial relationships among the transform-invariant features; (Fig.3 and Paragraph 55: “The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. … Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).” Fig.13 and Paragraph 168-169: “Fig.13, an interface 1350 is provided with information and graphical items as a result of the analysis performed … the interface 1350 includes a graphical item 1360 (e.g., floating window or dialog box), graphical item 1370, and graphical item 1380 that are superimposed over the captured image of the user's face. Each of the aforementioned graphical items includes information regarding a detected feature(s) or facial characteristic(s), such as a face shape (e.g., oval), indication of skin health, etc.”, The person of ordinary skill in the art would understand that the Lou teaches generating a GTIF product associated with the digital image of the physical object by using object recognition processes as scale-invariant feature transform (SIFT).)
based on, at least in part, the graph data structure, determining whether the particular GTIF product-token, generated by the computer-vision pipeline, associated with the digital image, matches a particular pair of a set of GTIF product-token pairs; (Paragraph 75: “the database 120 also stores data of products in a product table 316, which enables the product catalog system 124 to perform operations related to providing an augmented reality experience with respect to a product (e.g., a given physical item that may be available for purchase or sale). In an example, the product table 316 includes a directory (e.g., listing) of products and their associated product identifiers, which can be compared against product metadata provided by the product catalog system 124.”; Paragraph 108-109: “The product catalog system 124 can receive at least the product ID from the client device 102 (or the product catalog system 124 as discussed below), and perform a lookup, search, or select operation on the product table 316 to retrieve the product metadata from the database 120. … In an example, the product catalog system 124 can send a request message to a respective server for obtaining metadata related to a given physical item. The request message may include, for example, the product ID.” ; Fig.20 and Paragraph 202: “At operation 2006, the product identification module 704 extracts product metadata based on the determined object. In an embodiment, extracting the product metadata based on the determined object further includes comparing the identified object to a library of objects, each object from the library of objects including associated metadata with product information corresponding to a product, determining that the identified object matches a particular object from the library of object based at least in part on a confidence score being above a threshold value, and providing the associated metadata of the particular object.”)
wherein the set of GTIF product-token pairs comprises one or more pairs comprising known GTIF product-token and data determined for a user, a user device or a user event; (Paragraph 55-57: “The database 120 includes message data stored within a message table 314. The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of type, any entity regarding which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown). The entity graph 304 furthermore stores information regarding relationships and associations between entities. Such relationships may be social, professional (e.g., work at a common corporation or organization) interested-based or activity-based, … The database 120 also stores annotation data, in the example form of filters, in an annotation table 312. … Other types of filters include geolocation filters (also known as geo-filters) which may be presented to a sending user based on geographic location.”; Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”)
in response to determining that the particular GTIF product-token matches the particular pair, determining particular additional content based on the particular pair, (Fig. 20 and Paragraph 203-204: “At operation 2008, the product identification module 704 sends, to a server, the product metadata to determine second product metadata associated with the product metadata. The second product metadata includes media content from an external source, the external source being different than a messaging platform utilized by the client device. At operation 2010, the product identification module 704 receives, from the server, the second product metadata, the second product metadata including additional information related to the physical item.” Paragraph 110: “In response to the request message, the product catalog service system 602 can provide a response message to the product catalog system 124, which includes, in an example embodiment, the information from the database 620 in response to the request. The product catalog system 124 can store the received information into the database 120. Based at least on information retrieved from the database 120 related to the product, the client device 102 can provide for display (e.g., rendering on a UI of the messaging client application 104) an AR experience for the product.”)
wherein the particular additional content comprise hyperlink to the additional content; based on the hyperlinks, determining the additional content; (Figs. 10, 15-16 and 18 ; Paragraph 142: “an interface 1050 is provided showing additional information based at least in part on the product metadata described above, which may be received and displayed by the augmented reality content generator module 706 on the client device 102.As shown, production information 1060 is included (e.g., product description, price, manufacturer, vendor, selection graphical item to purchase the product). As further shown, media content 1065 (e.g., an embedded video), and additional media content 1070 are included in the interface 1050. In this example, the media content 1065 can be played directly within the interface 1050, whereas the additional media content 1070 may include respective descriptions and links to associated media content (e.g., respective video).”; Paragraphs 184-187; “information 1810 (e.g., a type of media content in the form of text) is included showing information based on a detected facial characteristic of the user (e.g., skin or face condition, eye condition, and the like), which can be based on the analysis performed by the product identification module 704 … A selectable graphical item 1812 (e.g., a button) is included to enable navigation to an external source (e.g., external web site) to view additional information, … media content 1830 includes a respective preview or graphical representation (e.g., image or short video) of associated media content (e.g., respective videos of tutorial or beauty routines, web links, etc.) which make demonstrate how to apply the product shown in the graphical item 1820”)
and causing, by the client application, an augmented reality (AR) overlay to be rendered on the user device; (Paragraphs 122: “the augmented reality content generator module 706 utilizes different object or facial detection processes to detect objects or a face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and render an AR experience based on the selected AR content generator from the carousel interface for display on a given client device (e.g., the client device 102” ; Paragraph 178: “an interface 1650 (which may be displayed by the augmented reality content generator module 706 on the client device 102) is shown with augmented reality content 1660 indicating the product that has been applied as augmented reality content 1670 on the captured image data corresponding to a representation of the user's face.”)
wherein the AR overlay is generated by compositing the additional content onto a live or static view of the physical object, using the spatial coordinates of the detected features as anchor points for the overlay, such that the additional content is visually registered to the physical object in the digital image. (Fig.7 and Paragraphs 122-124: “the augmented reality content generator module 706 utilizes different object or facial detection processes to detect objects or a face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and render an AR experience based on the selected AR content generator from the carousel interface for display on a given client device (e.g., the client device 102). … The augmented reality content generator module 706 performs rendering of content for display by the messaging client application 104 based on data provided by at least one of the aforementioned modules. For example, the augmented reality content generator module 706 performs various operations based on algorithms or techniques that correspond to animations or providing visual or auditory effects, based on the product information, to the received image data … the augmented reality content generator module 706 utilizes a graphical processing pipeline to perform graphical operations to render the message for display. The augmented reality content generator module 706 implements, in an example, an extensible rendering engine which supports multiple image processing operations corresponding to respective media overlays.” ; Figs. 13-15 and Paragraph 175: “an interface 1550 is provided, which may displayed by the augmented reality content generator module 706 on the client device 102 ... In this example, the position of the selectable graphical item 1560 corresponds to a “medium” setting (as the slider is in the middle), which results in augmented reality content 1564 to be rendered, based on this setting, on a portion of the user's face in the captured image data. As also shown, a selectable graphical item 1570 is provided, which when selected through user input can cause another interface to be displayed on the client device, such as a rendering of the user's face with the applied makeup in the form of the augmented reality content applied to the captured image data.”)
However, Luo does not disclose construct a graph data structure representing spatial relationships among the transform-invariant features of the physical object, wherein the graph data structure has a plurality of linked nodes that represent how some of the transform-invariant features of the physical object are linked with other transform-invariant features identified for the physical object;
Pope discloses receiving a digital image of a physical object; generating, by the client application, a particular graph of transform-invariant features (GTIF) product-token associated with the digital image of the physical object; (3. Approach: “Any image input to the system, whether for training or for recognition, is first represented as an attributed graph by a perceptual organization process. This image graph contains explicit information about the presence and spatial arrangement of significant features in the image. A series of image graphs, each an example of some object's appearance, is combined to form a model graph by a model learning procedure.”)
processing, by the client application, implementing feature detection and using a scale-invariant feature transform (SIFT), to extract a plurality of transform-invariant features, construct a graph data structure representing spatial relationships among the transform-invariant features of the physical object, wherein the graph data structure has a plurality of linked nodes that represent how some of the transform-invariant features of the physical object are linked with other transform-invariant features identified for the physical object; (3.1: Image and model graphs: “Once detected in an image, a feature is represented by a token that records the type of feature plus its image location, orientation, and scale. … These attributes are always expressed in a manner invariant with respect to translation, rotation, and scaling … An image graph records the hierarchy of tokens created from a single image: each node represents a token and the directed arcs represent composition and abstraction. … a model is represented like an image: as a graph whose nodes are feature tokens and whose arcs represent composition and abstraction relations among features. So that it can represent not just one appearance but a distribution of them the model graph contains additional statistical information”; 4. Experimental result: “Figure 2 shows the system learning a model for a cup, and figure 3 shows it recognizing the partly-occluded cup and two other objects in an image. The cup model contains 138 tokens and the image yields 308. About 30000 token matches were attempted in one minute to find an optimal match, which involves 61 token pairs.)
based on, at least in part, the graph data structure, determining whether the particular GTIF product-token associated with the digital image, matches a particular pair of a set of GTIF product-token pairs; (3.6: Modeling learning Procedure: “ From the first training image’s graph we select the connected subgraph containing the largest-scale token; this becomes the initial model. Each of its tokens has one attribute vector and is credited with one successful match. The next image’s graph is then compared with the model graph to find the best match.”; 4. Experiment Results: “Models were created for the two bottles using six training images of each. (Table 2 summarizes this process for one bottle.) Each model was then matched with two subimages from figure 3: one containing the identical object, the other containing its counterpart.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Luo by a model graph by a model learning procedure that is taught by Pope, to make the invention that systems and methods for authorizing rendering of objects in three-dimensional spaces; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving recognize an object in an image as well as enhancing the feature identified in image processing technique.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 10, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token comprises one or more digital hyperlinks indicating one or more locations in a storage device at which the particular additional content and identifiers of additional products have been stored; (Paragraph 172: “A selectable graphical item 1412 (e.g., a button) is included to enable adding this information regarding the detected face shape to a message. Media content 1420 includes respective descriptions and previews or graphical representations (e.g., images or short videos) of associated media content (e.g., respective videos of tutorial or beauty routines, web links, etc.). …the additional media content 1430 includes an embedded video that can be played within the interface 1400, without requiring the user to navigate to a. different interface or screen. In an embodiment, any or all of the aforementioned media content is provided (and stored) by an external source (e.g., a. third party) separate from the client device 102 and the messaging server system 108.”) wherein the particular additional content and the additional products are related to the physical object, have been constructed around the physical object, or are associated with circumstance related to an event related to the physical object or the digital image. (Paragraph 142: “an interface 1050 is provided showing additional information based at least in part on the product metadata described above, which may be received and displayed by the augmented reality content generator module 706 on the client device 102. … the media content 1065 can be played directly within the interface 1050, whereas the additional media content 1070 may include respective descriptions and links to associated media content (e.g., respective video). Additional information 1067 related to the product, or related to the media content (e.g., a description of the media content 1065) can be included in the interface 1050.”)
Regarding claim 11, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token is a complex data structure that is generated using advanced computer-based techniques that include one or more: encoding spatial representations of certain features identified in the digital image or determining a set of invariant features that are specific to the digital image; wherein the set of invariant features includes features that remain invariant of any 2D transformation performed on the features of the digital image. (Paragraph 62-66 ; Paragraph 118: “ The product identification module 704 utilizes different object detection processes to detect objects in the image, such as a physical item corresponding to a product that a user wants to extract product metadata from, or a physical indicator of identification (e.g., barcode) corresponding to the physical item. … optical character recognition (OCR) can be used as a primary image analysis technique or to enhance other processes. Features (e.g., shape, size, color and text) of the image can be extracted. image processing processes may include sub-processes such as … segmentation, blob extraction, pattern recognition, barcode and data matrix code reading, gauging (measuring object dimensions), positioning, edge detection, color analysis, filtering (e.g. morphological filtering) and template matching (finding, matching, or counting specific patterns).”)
Regarding claim 12, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token pair, of the set of GTIF product-token pairs, represents one or more of: one or more of relationships between a plurality of transform-invariant features identified for the digital image, or one or more relationships between the plurality of transform-invariant features identified for the digital image and other transform-invariant features identified for other products. (Paragraph 107: “he product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, Universal Product Code (UPC) codes, QR codes … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”; Paragraphs 62-66 and 118)
Regarding claim 13, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the digital image has a plurality transform invariant features and a corresponding plurality of GTIF product-tokens; wherein a GTIF product-token is used to determine whether the GTIF product-token matches a particular pair of the set of GTIF product-token pairs; (Paragraph 75: “The database 120 also stores data of products in a product table 316, which enables the product catalog system 124 to perform operations related to providing an augmented reality experience with respect to a product (e.g., a given physical item that may be available for purchase or sale). In an example, the product table 316 includes a directory (e.g., listing) of products and their associated product identifiers, which can be compared against product metadata provided by the product catalog system 124”) wherein a GTIF product-token pair comprises additional context data that include one or more of: location data determined based on GPS location data obtained from one or more of: the location of the user device, a photo, an address of an event, or an address of customers or users; social relationship data of a creator or a recipient of the digital image; or time based data determined based on one or more of: a time of an event, a time when a photo was taken, or a time when a message was sent. (Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, Universal Product Code (UPC) codes, QR codes … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”)
Regarding claim 14, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the determining that the particular GTIF product-token matches the particular pair is a search that requires comparisons between non-directed graphs having a plurality of nodes, wherein the nodes represent transform invariant features, wherein a time for comparison performed as a series of instructions on computing machinery increases based on a number of comparisons, and wherein a number of transform invariant features exceeds practical limits of user interaction time; wherein a GTIF product-token is generated using one or more of: a scale-invariant feature transform feature recognition method (SIFT), a simultaneous localization and mapping feature recognition method (SLAM), or a speed up robust features feature recognition method (SURF). (Paragraph 108; Paragraph 167: “other object recognition processes can be used. … Other techniques include feature-based techniques, such as may utilize interpretation trees, geometric hashing, or invariance analysis. Moreover, techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.”)
Regarding claim 15, Luo discloses a custom product computer system generator (Paragraph 240: “ FIG. 25 is a block diagram illustrating components of a machine 2500,) comprising: a memory unit; (Fig.25: memory/storage 2506) one or more processors; (Fig.25: processor 2504) and a custom product computer storing one or more instructions, which, when executed by one or more processors, (Paragraph 240: “FIG. 25 shows a diagrammatic representation of the machine 2500 in the example form of a computer system, within which instructions 2510 (e.g., software, a program, an application, an apples, an app, or other executable code) for causing the machine 2500 to perform any one or more of the methodologies discussed herein may be executed.”) cause the one or more processors to perform:
using a client application executing on a user device, receiving a digital image of a physical object; (Fig. 7 and Paragraph 117: “ Paragraph 117: “The image data receiving module 702 receives images captured by a client device 102. For example, an image is a photograph captured by an optical sensor (e.g., camera) of the client device 102. An image includes one or more real-world features, such a user's face, or a physical object(s) detected in the image.”; Fig. 20 ; Paragraph 199: “At operation 2002, the image data receiving module 702 receives image data including a representation of a physical item. In an example, the physical item is a beauty product”)
generating, by the client application, a particular graph of transform-invariant features (GTIF) product-token associated with the digital image of the physical object; (Fig. 7; Paragraph 118-120: “The product identification module 704 utilizes different object detection processes to detect objects in the image, such as a physical item corresponding to a product that a user wants to extract product metadata from, or a physical indicator of identification (e.g., barcode) corresponding to the physical item. … the product identification module 704, in an example embodiment, utilizes a set of classifiers that classify an image received from a camera of a mobile computing device into one or more classes. In an embodiment, an example set of image classifier determines whether the image includes a physical identification indicator containing text, barcode pattern(s), or QR code pattern(s), and the like. … The product identification module 704 can perform operations (e.g., a process) for extracting product metadata from a recognized object corresponding to a physical identification indicator of the physical item in the image,”; Figs. 9-20 and Paragraph 134-145; Paragraph 202: “At operation 2006, the product identification module 704 extracts product metadata based on the determined object.”)
processing, by the client application, the digital image using a computer-vision pipeline, implementing feature detection and using a scale-invariant feature transform (SIFT), to extract a plurality of transform-invariant features, (Paragraph 120: “the product identification module 704 can perform operations (e.g., a process) for extracting product metadata from a recognized object corresponding to a physical identification indicator of the physical item in the image,”; Paragraph 160-167: “Fig.13 example interfaces for scanning a user's face and presenting various information based on detected features or facial characteristics, and the focal region 1330 can be analyzed by the product identification module 704 or other component of the client device 102) using techniques described below (read as “the digital image using a computer- vision pipeline”) to attempt to identity feature points corresponding to one or more facial features of the user's face and other facial characteristics … the aforementioned detection techniques can determine the facial features and facial characteristics using a number of different techniques … other object recognition processes can be used. … Moreover, techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.”) and construct a graph data structure representing spatial relationships among the transform-invariant features; (Fig.3 and Paragraph 55: “The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. … Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).” Fig.13 and Paragraph 168-169: “Fig.13, an interface 1350 is provided with information and graphical items as a result of the analysis performed … the interface 1350 includes a graphical item 1360 (e.g., floating window or dialog box), graphical item 1370, and graphical item 1380 that are superimposed over the captured image of the user's face. Each of the aforementioned graphical items includes information regarding a detected feature(s) or facial characteristic(s), such as a face shape (e.g., oval), indication of skin health, etc.”, The person of ordinary skill in the art would understand that the Lou teaches generating a GTIF product associated with the digital image of the physical object by using object recognition processes as scale-invariant feature transform (SIFT).)
based on, at least in part, the graph data structure, determining whether the particular GTIF product-token, generated by the computer-vision pipeline, associated with the digital image, matches a particular pair of a set of GTIF product-token pairs; (Paragraph 75: “the database 120 also stores data of products in a product table 316, which enables the product catalog system 124 to perform operations related to providing an augmented reality experience with respect to a product (e.g., a given physical item that may be available for purchase or sale). In an example, the product table 316 includes a directory (e.g., listing) of products and their associated product identifiers, which can be compared against product metadata provided by the product catalog system 124.”; Paragraph 108-109: “The product catalog system 124 can receive at least the product ID from the client device 102 (or the product catalog system 124 as discussed below), and perform a lookup, search, or select operation on the product table 316 to retrieve the product metadata from the database 120. … In an example, the product catalog system 124 can send a request message to a respective server for obtaining metadata related to a given physical item. The request message may include, for example, the product ID.” ; Fig.20 and Paragraph 202: “At operation 2006, the product identification module 704 extracts product metadata based on the determined object. In an embodiment, extracting the product metadata based on the determined object further includes comparing the identified object to a library of objects, each object from the library of objects including associated metadata with product information corresponding to a product, determining that the identified object matches a particular object from the library of object based at least in part on a confidence score being above a threshold value, and providing the associated metadata of the particular object.”)
wherein the set of GTIF product-token pairs comprises one or more pairs comprising known GTIF product-token and data determined for a user, a user device or a user event; (Paragraph 55-57: “The database 120 includes message data stored within a message table 314. The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of type, any entity regarding which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown). The entity graph 304 furthermore stores information regarding relationships and associations between entities. Such relationships may be social, professional (e.g., work at a common corporation or organization) interested-based or activity-based, … The database 120 also stores annotation data, in the example form of filters, in an annotation table 312. … Other types of filters include geolocation filters (also known as geo-filters) which may be presented to a sending user based on geographic location.”; Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”)
in response to determining that the particular GTIF product-token matches the particular pair, determining particular additional content based on the particular pair, (Fig. 20 and Paragraph 203-204: “At operation 2008, the product identification module 704 sends, to a server, the product metadata to determine second product metadata associated with the product metadata. The second product metadata includes media content from an external source, the external source being different than a messaging platform utilized by the client device. At operation 2010, the product identification module 704 receives, from the server, the second product metadata, the second product metadata including additional information related to the physical item.” Paragraph 110: “In response to the request message, the product catalog service system 602 can provide a response message to the product catalog system 124, which includes, in an example embodiment, the information from the database 620 in response to the request. The product catalog system 124 can store the received information into the database 120. Based at least on information retrieved from the database 120 related to the product, the client device 102 can provide for display (e.g., rendering on a UI of the messaging client application 104) an AR experience for the product.”)
wherein the particular additional content comprise hyperlink to the additional content; based on the hyperlinks, determining the additional content; (Figs. 10, 15-16 and 18 ; Paragraph 142: “an interface 1050 is provided showing additional information based at least in part on the product metadata described above, which may be received and displayed by the augmented reality content generator module 706 on the client device 102.As shown, production information 1060 is included (e.g., product description, price, manufacturer, vendor, selection graphical item to purchase the product). As further shown, media content 1065 (e.g., an embedded video), and additional media content 1070 are included in the interface 1050. In this example, the media content 1065 can be played directly within the interface 1050, whereas the additional media content 1070 may include respective descriptions and links to associated media content (e.g., respective video).”; Paragraphs 184-187; “information 1810 (e.g., a type of media content in the form of text) is included showing information based on a detected facial characteristic of the user (e.g., skin or face condition, eye condition, and the like), which can be based on the analysis performed by the product identification module 704 … A selectable graphical item 1812 (e.g., a button) is included to enable navigation to an external source (e.g., external web site) to view additional information, … media content 1830 includes a respective preview or graphical representation (e.g., image or short video) of associated media content (e.g., respective videos of tutorial or beauty routines, web links, etc.) which make demonstrate how to apply the product shown in the graphical item 1820”) and
causing, by the client application, an augmented reality (AR) overlay to be rendered on the user device; (Paragraphs 122: “the augmented reality content generator module 706 utilizes different object or facial detection processes to detect objects or a face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and render an AR experience based on the selected AR content generator from the carousel interface for display on a given client device (e.g., the client device 102” ; Paragraph 178: “an interface 1650 (which may be displayed by the augmented reality content generator module 706 on the client device 102) is shown with augmented reality content 1660 indicating the product that has been applied as augmented reality content 1670 on the captured image data corresponding to a representation of the user's face.”)
wherein the AR overlay is generated by compositing the additional content onto a live or static view of the physical object, using the spatial coordinates of the detected features as anchor points for the overlay, such that the additional content is visually registered to the physical object in the digital image. (Fig.7 and Paragraphs 122-124: “the augmented reality content generator module 706 utilizes different object or facial detection processes to detect objects or a face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and render an AR experience based on the selected AR content generator from the carousel interface for display on a given client device (e.g., the client device 102). … The augmented reality content generator module 706 performs rendering of content for display by the messaging client application 104 based on data provided by at least one of the aforementioned modules. For example, the augmented reality content generator module 706 performs various operations based on algorithms or techniques that correspond to animations or providing visual or auditory effects, based on the product information, to the received image data … the augmented reality content generator module 706 utilizes a graphical processing pipeline to perform graphical operations to render the message for display. The augmented reality content generator module 706 implements, in an example, an extensible rendering engine which supports multiple image processing operations corresponding to respective media overlays.” ; Figs. 13-15 and Paragraph 175: “an interface 1550 is provided, which may displayed by the augmented reality content generator module 706 on the client device 102 ... In this example, the position of the selectable graphical item 1560 corresponds to a “medium” setting (as the slider is in the middle), which results in augmented reality content 1564 to be rendered, based on this setting, on a portion of the user's face in the captured image data. As also shown, a selectable graphical item 1570 is provided, which when selected through user input can cause another interface to be displayed on the client device, such as a rendering of the user's face with the applied makeup in the form of the augmented reality content applied to the captured image data.”)
However, Luo does not disclose construct a graph data structure representing spatial relationships among the transform-invariant features of the physical object, wherein the graph data structure has a plurality of linked nodes that represent how some of the transform-invariant features of the physical object are linked with other transform-invariant features identified for the physical object;
Pope discloses receiving a digital image of a physical object; generating, by the client application, a particular graph of transform-invariant features (GTIF) product-token associated with the digital image of the physical object; (3. Approach: “Any image input to the system, whether for training or for recognition, is first represented as an attributed graph by a perceptual organization process. This image graph contains explicit information about the presence and spatial arrangement of significant features in the image. A series of image graphs, each an example of some object's appearance, is combined to form a model graph by a model learning procedure.”)
processing, by the client application, implementing feature detection and using a scale-invariant feature transform (SIFT), to extract a plurality of transform-invariant features, construct a graph data structure representing spatial relationships among the transform-invariant features of the physical object, wherein the graph data structure has a plurality of linked nodes that represent how some of the transform-invariant features of the physical object are linked with other transform-invariant features identified for the physical object; (3.1: Image and model graphs: “Once detected in an image, a feature is represented by a token that records the type of feature plus its image location, orientation, and scale. … These attributes are always expressed in a manner invariant with respect to translation, rotation, and scaling … An image graph records the hierarchy of tokens created from a single image: each node represents a token and the directed arcs represent composition and abstraction. … a model is represented like an image: as a graph whose nodes are feature tokens and whose arcs represent composition and abstraction relations among features. So that it can represent not just one appearance but a distribution of them the model graph contains additional statistical information”; 4. Experimental result: “Figure 2 shows the system learning a model for a cup, and figure 3 shows it recognizing the partly-occluded cup and two other objects in an image. The cup model contains 138 tokens and the image yields 308. About 30000 token matches were attempted in one minute to find an optimal match, which involves 61 token pairs.)
based on, at least in part, the graph data structure, determining whether the particular GTIF product-token associated with the digital image, matches a particular pair of a set of GTIF product-token pairs; (3.6: Modeling learning Procedure: “ From the first training image’s graph we select the connected subgraph containing the largest-scale token; this becomes the initial model. Each of its tokens has one attribute vector and is credited with one successful match. The next image’s graph is then compared with the model graph to find the best match.”; 4. Experiment Results: “Models were created for the two bottles using six training images of each. (Table 2 summarizes this process for one bottle.) Each model was then matched with two subimages from figure 3: one containing the identical object, the other containing its counterpart.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Luo by a model graph by a model learning procedure that is taught by Pope, to make the invention that systems and methods for authorizing rendering of objects in three-dimensional spaces; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving recognize an object in an image as well as enhancing the feature identified in image processing technique.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 17, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token comprises one or more digital hyperlinks indicating one or more locations in a storage device at which the particular additional content and identifiers of additional products have been stored; (Paragraph 172: “A selectable graphical item 1412 (e.g., a button) is included to enable adding this information regarding the detected face shape to a message. Media content 1420 includes respective descriptions and previews or graphical representations (e.g., images or short videos) of associated media content (e.g., respective videos of tutorial or beauty routines, web links, etc.). …the additional media content 1430 includes an embedded video that can be played within the interface 1400, without requiring the user to navigate to a. different interface or screen. In an embodiment, any or all of the aforementioned media content is provided (and stored) by an external source (e.g., a. third party) separate from the client device 102 and the messaging server system 108.”) wherein the particular additional content and the additional products are related to the physical object, have been constructed around the physical object, or are associated with circumstance related to an event related to the physical object or the digital image. (Paragraph 142: “an interface 1050 is provided showing additional information based at least in part on the product metadata described above, which may be received and displayed by the augmented reality content generator module 706 on the client device 102. … the media content 1065 can be played directly within the interface 1050, whereas the additional media content 1070 may include respective descriptions and links to associated media content (e.g., respective video). Additional information 1067 related to the product, or related to the media content (e.g., a description of the media content 1065) can be included in the interface 1050.”)
Regarding claim 18, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token is a complex data structure that is generated using advanced computer-based techniques that include one or more: encoding spatial representations of certain features identified in the digital image or determining a set of invariant features that are specific to the digital image; wherein the set of invariant features includes features that remain invariant of any 2D transformation performed on the features of the digital image. (Paragraph 62-66 ; Paragraph 118: “ The product identification module 704 utilizes different object detection processes to detect objects in the image, such as a physical item corresponding to a product that a user wants to extract product metadata from, or a physical indicator of identification (e.g., barcode) corresponding to the physical item. … optical character recognition (OCR) can be used as a primary image analysis technique or to enhance other processes. Features (e.g., shape, size, color and text) of the image can be extracted. image processing processes may include sub-processes such as … segmentation, blob extraction, pattern recognition, barcode and data matrix code reading, gauging (measuring object dimensions), positioning, edge detection, color analysis, filtering (e.g. morphological filtering) and template matching (finding, matching, or counting specific patterns).”)
Regarding claim 19, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses a GTIF product-token pair, of the set of GTIF product-token pairs, represents one or more of: one or more of relationships between a plurality of transform-invariant features identified for the digital image, or one or more relationships between the plurality of transform-invariant features identified for the digital image and other transform-invariant features identified for other products. (Paragraph 107: “he product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, Universal Product Code (UPC) codes, QR codes … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”; Paragraphs 62-66 and 118)
Regarding claim 20, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the digital image has a plurality transform invariant features and a corresponding plurality of GTIF product- tokens; wherein a GTIF product-token is used to determine whether the GTIF product-token matches a particular pair of the set of GTIF product-token pairs; (Paragraph 75: “The database 120 also stores data of products in a product table 316, which enables the product catalog system 124 to perform operations related to providing an augmented reality experience with respect to a product (e.g., a given physical item that may be available for purchase or sale). In an example, the product table 316 includes a directory (e.g., listing) of products and their associated product identifiers, which can be compared against product metadata provided by the product catalog system 124”) wherein a GTIF product-token pair comprises additional context data that include one or more of: location data determined based on GPS location data obtained from one or more of: the location of the user device, a photo, an address of an event, or an address of customers or users; social relationship data of a creator or a recipient of the digital image; or time based data determined based on one or more of: a time of an event, a time when a photo was taken, or a time when a message was sent; (Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, Universal Product Code (UPC) codes, QR codes … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”) wherein the determining that the particular GTIF product-token matches the particular pair is a search that requires comparisons between non-directed graphs having a plurality of nodes, wherein the nodes represent transform invariant features, wherein a time for comparison performed as a series of instructions on computing machinery increases based on a number of comparisons, and wherein a number of transform invariant features exceeds practical limits of user interaction time; wherein a GTIF product-token is generated using one or more of: a scale-invariant feature transform feature recognition method (SIFT), a simultaneous localization and mapping feature recognition method (SLAM), or a speed up robust features feature recognition method (SURF). (Paragraph 108 ; Paragraph 167: “other object recognition processes can be used. … Other techniques include feature-based techniques, such as may utilize interpretation trees, geometric hashing, or invariance analysis. Moreover, techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.”)
Claim(s) 2, 9 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (U.S. 20210312533 A1; Luo), in view of Pope, Arthur et al (“Learning object recognition models from images.”; Pope), and in further view of Collart (U.S. 20200104522 A1).
Regarding claim 2, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the digital image is obtained by scanning the physical object using a scanner; wherein the scanner is any one of: a standalone camera communicatively coupled to the user device, a camera installed in the user device, or a standalone scanner communicatively coupled to the user device; (Paragraph 106: “the client device 102 may provide image data including a representation of a physical item (e.g., captured using a camera provided by the client device 102) including an identification indicator (e.g., a physical barcode, etc.) of the physical item”) wherein the user device is any one of: a smartphone, an iPad, a laptop, or a PDA. (Paragraph 111: “the client device 102 as described above in FIG. 6 may be, for example, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera), a tablet device, a wearable device such as a watch, a band, a headset, and the like, or any other appropriate device.”) wherein the computer-vision pipeline further implements speeded up robust features (SURF), (Paragraph 167: “other object recognition processes can be used. … techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.)
wherein the set of GTIF product-token pairs comprises one or more of: a pair comprising a known GTIF product-token and a location data determined for a location of a user device, a pair comprising known GTIF product-token associated with a user of the user device and one or more social relationships defined for the user, a pair comprising known time based data associated with one or more events defined for the user and the one or more events, a pair comprising a GTIF product token and a representation of a physical object detected by a camera or sensors and communicated to the user device, or a pair comprising a GTIF product token and a representation of a digital object provided by the user device. Paragraph 55-57: “The database 120 includes message data stored within a message table 314. The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of type, any entity regarding which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown). The entity graph 304 furthermore stores information regarding relationships and associations between entities. Such relationships may be social, professional (e.g., work at a common corporation or organization) interested-based or activity-based, … The database 120 also stores annotation data, in the example form of filters, in an annotation table 312. … Other types of filters include geolocation filters (also known as geo-filters) which may be presented to a sending user based on geographic location.”; Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”)
However, Luo as modified by Pope, does not disclose simultaneous localization and mapping (SLAM) algorithm
Collart discloses the computer-vision pipeline further implements simultaneous localization and mapping (SLAM) algorithm (Paragraphs 146-147: “An xR system may also utilize a variety of technologies for capturing detail of a location, determining viewing angle/direction, and establishing proximity such as techniques to create a 3D scan (e.g., RGB camera plus a depth camera, single RGB camera and computer vision, visual inertial odometry, or a pre-existing online database of 3D scan data). … his may be possible using a visual mapping and localization system (SLAM) that can parse a visual scene and track position in a highly accurate manner.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Luo and Pope by including a visual mapping and localization system (SLAM) that is taught by Collart, to make the invention that systems and methods for authorizing rendering of objects in three-dimensional spaces; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the accuracy of track position as well as constructing highly detailed map data.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 9, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the digital image is obtained by scanning the physical object using a scanner; wherein the scanner is any one of: a standalone camera communicatively coupled to the user device, a camera installed in the user device, or a standalone scanner communicatively coupled to the user device; (Paragraph 106: “the client device 102 may provide image data including a representation of a physical item (e.g., captured using a camera provided by the client device 102) including an identification indicator (e.g., a physical barcode, etc.) of the physical item”) wherein the user device is any one of: a smartphone, an iPad, a laptop, or a PDA. (Paragraph 111: “the client device 102 as described above in FIG. 6 may be, for example, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera), a tablet device, a wearable device such as a watch, a band, a headset, and the like, or any other appropriate device.”) wherein the computer-vision pipeline further implements speeded up robust features (SURF), (Paragraph 167: “other object recognition processes can be used. … techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.)
wherein the set of GTIF product-token pairs comprises one or more of: a pair comprising a known GTIF product-token and a location data determined for a location of a user device, a pair comprising known GTIF product-token associated with a user of the user device and one or more social relationships defined for the user, a pair comprising known time based data associated with one or more events defined for the user and the one or more events, a pair comprising a GTIF product token and a representation of a physical object detected by a camera or sensors and communicated to the user device, or a pair comprising a GTIF product token and a representation of a digital object provided by the user device. Paragraph 55-57: “The database 120 includes message data stored within a message table 314. The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of type, any entity regarding which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown). The entity graph 304 furthermore stores information regarding relationships and associations between entities. Such relationships may be social, professional (e.g., work at a common corporation or organization) interested-based or activity-based, … The database 120 also stores annotation data, in the example form of filters, in an annotation table 312. … Other types of filters include geolocation filters (also known as geo-filters) which may be presented to a sending user based on geographic location.”; Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”)
However, Luo, as modified by Pope, does not disclose simultaneous localization and mapping (SLAM) algorithm
Collart discloses the computer-vision pipeline further implements simultaneous localization and mapping (SLAM) algorithm (Paragraphs 146-147: “An xR system may also utilize a variety of technologies for capturing detail of a location, determining viewing angle/direction, and establishing proximity such as techniques to create a 3D scan (e.g., RGB camera plus a depth camera, single RGB camera and computer vision, visual inertial odometry, or a pre-existing online database of 3D scan data). … his may be possible using a visual mapping and localization system (SLAM) that can parse a visual scene and track position in a highly accurate manner.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Luo and Pope by including a visual mapping and localization system (SLAM) that is taught by Collart, to make the invention that systems and methods for authorizing rendering of objects in three-dimensional spaces; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the accuracy of track position as well as constructing highly detailed map data.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 16, Luo, as modified by Pope, discloses all the claims invention. Luo further discloses the digital image is obtained by scanning the physical object using a scanner; wherein the scanner is any one of: a standalone camera communicatively coupled to the user device, a camera installed in the user device, or a standalone scanner communicatively coupled to the user device; (Paragraph 106: “the client device 102 may provide image data including a representation of a physical item (e.g., captured using a camera provided by the client device 102) including an identification indicator (e.g., a physical barcode, etc.) of the physical item”) wherein the user device is any one of: a smartphone, an iPad, a laptop, or a PDA. (Paragraph 111: “the client device 102 as described above in FIG. 6 may be, for example, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera), a tablet device, a wearable device such as a watch, a band, a headset, and the like, or any other appropriate device.”) wherein the computer-vision pipeline further implements speeded up robust features (SURF), (Paragraph 167: “other object recognition processes can be used. … techniques such as scale-invariant feature transform (SIFT) or speeded up robust features (SURF) techniques can also be used within the scope of the subject technology.)
wherein the user device is any one of: a smartphone, an iPad, a laptop, or a PDA. (Paragraph 111: “the client device 102 as described above in FIG. 6 may be, for example, a portable computing device such as a laptop computer, a smartphone, a peripheral device (e.g., a digital camera), a tablet device, a wearable device such as a watch, a band, a headset, and the like, or any other appropriate device.”)
wherein the set of GTIF product-token pairs comprises one or more of: a pair comprising a known GTIF product-token and a location data determined for a location of a user device, a pair comprising known GTIF product-token associated with a user of the user device and one or more social relationships defined for the user, a pair comprising known time based data associated with one or more events defined for the user and the one or more events, a pair comprising a GTIF product token and a representation of a physical object detected by a camera or sensors and communicated to the user device, or a pair comprising a GTIF product token and a representation of a digital object provided by the user device. Paragraph 55-57: “The database 120 includes message data stored within a message table 314. The entity table 302 stores entity data, including an entity graph 304. Entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of type, any entity regarding which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown). The entity graph 304 furthermore stores information regarding relationships and associations between entities. Such relationships may be social, professional (e.g., work at a common corporation or organization) interested-based or activity-based, … The database 120 also stores annotation data, in the example form of filters, in an annotation table 312. … Other types of filters include geolocation filters (also known as geo-filters) which may be presented to a sending user based on geographic location.”; Paragraph 107: “the product catalog system 124 can receive the aforementioned extracted information from the client device 102, and determine additional product information (e.g., product metadata). Additional product metadata of a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) including information that respective manufacturers have maintained regarding individual products including, but not limited to barcodes, … Other signals that may be utilized can include location information (e.g., GPS coordinates to determine a particular reseller or retail, or geographic region corresponding to the physical item), network information (e.g., Wi-Fi network), etc.”)
However, Luo, as modified by Pope, does not disclose simultaneous localization and mapping (SLAM) algorithm
Collart discloses the computer-vision pipeline further implements simultaneous localization and mapping (SLAM) algorithm (Paragraphs 146-147: “An xR system may also utilize a variety of technologies for capturing detail of a location, determining viewing angle/direction, and establishing proximity such as techniques to create a 3D scan (e.g., RGB camera plus a depth camera, single RGB camera and computer vision, visual inertial odometry, or a pre-existing online database of 3D scan data). … his may be possible using a visual mapping and localization system (SLAM) that can parse a visual scene and track position in a highly accurate manner.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Luo and Pope by including a visual mapping and localization system (SLAM) that is taught by Collart, to make the invention that systems and methods for authorizing rendering of objects in three-dimensional spaces; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the accuracy of track position as well as constructing highly detailed map data.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Relevant Prior Art Directed to State of Art
Neumeier et al (U.S. 20170019719 A1), “Detection Of Common Media Segments”, teaches about Systems, methods, and computer-program products for identifying a media content stream when the media content stream is playing an unscheduled media segment. A computing device may receive a plurality of media content streams, where at least two of the plurality of media content streams concurrently includes a same unscheduled media segment. The computing device may determine that the media display device is playing the unscheduled media segment by examining the media content available at the current time in each of the plurality of media content streams. The computing device may determine identification information from the media content included in the media content stream and contextually-related content,. The computing device may display the media content stream and the contextually-related content after the unscheduled media segment has been played.
King et al (U.S. 20110145068 A1), “Associating Rendered Advertisements With Digital Content”, teaches about a system and method for associating rendered advertisements with digital content is described. In some examples, the system receives an image of a rendered advertisement, information associated with digital content, and information associating the rendered advertisement with the digital content via a web portal.
Galuten (U.S. 20230075182 A1), “System and Methods for Managing Content from Creation To Consumption”, teaches about systems and methods are provided to facilitate creation of works of any type, to collaborate with co-creators, to make that work available in the marketplace, and to provide for an architecture where the identities of all the creators involved may be bound to associated objects and/or identifiers associated with them. Being securely bound to certain identities and the identifiers, creative works may be distributed, and the associated credits and contractual obligations may remain with them and the associated remunerations and obligations may be respected and fulfilled.
Goldson et al (U.S. 20210279305 A1), “Tokenized Media Content Management”, teaches about systems and methods may be implemented to create, manage and share one or more content items, along with metadata or other related files associated with those content items. The system may be implemented to create a container to contain one or more content items and associated metadata. The system may be further implemented to verify the completeness of metadata files associated with content items in the containers, alert appropriate users if specified metadata files are missing from a container, and allow users to update the metadata to complete the containers.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Duy A Tran whose telephone number is (571)272-4887. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL R MISTRY can be reached at (313)-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY TRAN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674