Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/3/ 2026 has been entered.
Response to Amendment
Applicant's arguments and REMARKS filed 03/03/2026 regarding independent claims have been fully considered but they are not persuasive. Claims 1,3, 9, 11, 15, 17 have been amended. In summary, claims 1-20 are pending in application.
Claim Rejections under 35 U.S.C. § 112(b):
Applicant has amended claims 3, 11 and 17 to overcome the previous rejection. The rejection of claim 3, 11, 17 under 35 U.S.C. § 112(b) has been withdrawn.
Claim Rejections - 35 USC § 103
Applicant’s arguments with respect to independent claims have been considered but are moot because the rejection has been modified to address the newly added limitations. The Examiner now relies on the references Grant for the argued limitation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1-4, 6-7, 9-12, 14-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Trim et al, U.S Patent Application Publication No 2020/0387537 (“Trim”) in view of Goodrich et al, U.S Patent Application Publication No.2022/0197027 (“Goodrich”) further in view of Tiutiunnik et al. U.S Patent No.11430211 (“Tiutiunnik”) further in view of Grant et al., U.S Patent Application Publication No. 20200401802 (“Grant”)
Regarding independent claim 1, Trim teaches a method comprising :at a computing system including non-transitory memory and one or more processors ([0089] Computer system 800 may include communications fabric 802, which provides for communications between one or more processors 803, memory 805, persistent storage 806, communications unit 811, and one or more input/output (I/O) interfaces 815. Communications fabric 802 can be implemented with any architecture designed for passing data and/or controlling information between processors 803 (such as microprocessors, communications and network processors, etc.), memory 805, external devices 817, and any other hardware components within a system. For example, communications fabric 802 can be implemented with one or more buses.”), wherein the computing system is communicatively coupled to a display device and one or more input devices via a communication interface ([0092] Communications unit 811 provides for communications with other computer systems or devices via a network (e.g., communication network 150). In the exemplary embodiment, communications unit 811 may include network adapters or interfaces such as a TCP/IP adapter cards, wireless Wi-Fi interface cards, 3G, 4G, or 5G wireless interface cards or other wired or wireless communication links. The communication network 150 can comprise, for example, copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, edge servers and/or other infrastructure and resources 209 which may be part of the radio nodes 211, cloud nodes 213 and network nodes 215 which create the communication network 150. Software and data used to practice embodiments of the present invention can be downloaded to each of the computer systems operating in computing environment 100 or computer system 800 through communications unit 811 (e.g., via the Internet, a local area network or other wide area network). From communications unit 811, the software and data can be loaded onto persistent storage 806. [0093] One or more I/O interfaces 815 may allow for input and output of data with other devices that may be connected to computer system 800. For example, I/O interface 815 can provide a connection to one or more external devices 817 such as one or more internet-of-things devices, recording devices such as an audio system, camera systems, one or more sensor device(s), input devices such as a keyboard, computer mouse, touch screen, virtual keyboard, touch pad, pointing device, or other human interface devices. External devices 817 can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. I/O interface 815 may connect to human-readable display 818. Human-readable display 818 provides a mechanism to display data to a user and can be, for example, a computer monitor or screen. Human-readable display 818 can also be an incorporated display and may function as a touch screen, such as a built-in display of a tablet computer.”):
obtaining an electronic message addressed to a user of the computing system in a physical environment displaying a current field-of-view (FOV) of a physical environment on the display device, wherein the electronic message is sent from a sender to the user in a messaging application and includes a text portion indicated by the sender as associated with a real-world object (see at least [0056] as shown in Fig 6a-6b “Referring to the drawings, FIG. 6a-6b describes a second example of medical services 117 analyzing streaming inputs entering the personal medical channel 227 and tagging the data of the streaming inputs with situational specific tags that provide additional context to the streaming inputs of data, allowing the end user suffering from a visual impairment to otherwise understand the context of the data being received on the end user's device 101. As depicted in the example of FIG. 6a, an end user 601 operating an end user device 101 engages in a text messaging conversation being transmitted between a second user via the personal medical channel 227 of the communications network 150. In FIG. 6a, a description of a vehicle is provided to the end user 601 via text messaging data, while an image of the vehicle is provided to the end user in FIG. 6b. Embodiments of the medical services 117 may analyze the text messaging data and image data depicting the vehicle within the context of the conversation topic between the end user 601 and the second user by monitoring the communication data transmitted using the personal medical channel 227.” 0059] Referring to the drawings, FIGS. 6c-6d provide an example of an embodiment of vision assistance that may be outputted by the medical services 117 to an end user device 101. In the example of FIG. 6c-6d, IoT devices, such as a smart display, may utilize augmented reality, vision recognition and other imaging techniques to provide descriptive overlays comprising situation-specific tags 603 onto the smart display. The overlays may be positioned onto or adjacent to physical objects viewed through the display 818 based on where the end user's gaze is directed. The one or more physical objects within the field of vision of the smart display or other types of human-readable displays 818 may be tagged with a situational-specific tag as shown in FIG. 6d. In the example of FIG. 6c-6d, can be a continuation of the example described above for FIG. 6a-6b. Continuing from the FIG. 6a-6b example, an end user equipped with a smart display or another computing device for viewing one or more physical objects, may proceed to change locations in order to find the second user's vehicle as described by the messaging data in FIG. 6a and/or the image provided in FIG. 6b. As the end user views the second user's vehicle through the smart device or other augmented reality capable device, the situation-specific tag 603 appears on the display 818 as shown in FIG. 6d, providing a tagged description and/or indicator identifying the vehicle so the end user knows which vehicle matches the one identified within the context of the conversation between the end user and the second user, based on vehicle description, image of the vehicle and/or additional streaming inputs entering through the personal medical channel from one or more additional data sources 119.”);
in response to determining that the electronic message includes the text portion indicated by the sender as associated with the real-world object, determining whether the current FOV of the physical environment includes the real- world object corresponding to the text portion of the electronic message (see at least [0057] as shown in Fig. 6a-6c “Analyzing the text message descriptions provided in FIG. 6a along with the image transmitted over the communication network 150, as shown in FIG. 6b, the medical services 117 may be able to infer which car being described to the end user is the second user's vehicle within the image. The identified vehicle based on the context of the transmitted messages and images may be tagged with situational-specific tag 603 as shown in FIG. 6b. The situational-specific tag 603 may be inserted in-line with the data being transmitted over the communication network 150 and outputted to the end user device 101 alongside the image data depicting the vehicle. As shown in FIG. 6b, the situational-specific tag 603 is overlaid onto the image displayed by the end user device 101, pointing to the second user's vehicle and indicating a description of the vehicle in this example (i.e. “Driver's blue sedan”).- [0059] Referring to the drawings, FIGS. 6c-6d provide an example of an embodiment of vision assistance that may be outputted by the medical services 117 to an end user device 101. In the example of FIG. 6c-6d, IoT devices, such as a smart display, may utilize augmented reality, vision recognition and other imaging techniques to provide descriptive overlays comprising situation-specific tags 603 onto the smart display. The overlays may be positioned onto or adjacent to physical objects viewed through the display 818 based on where the end user's gaze is directed. The one or more physical objects within the field of vision of the smart display or other types of human-readable displays 818 may be tagged with a situational-specific tag as shown in FIG. 6d. In the example of FIG. 6c-6d, can be a continuation of the example described above for FIG. 6a-6b. Continuing from the FIG. 6a-6b example, an end user equipped with a smart display or another computing device for viewing one or more physical objects, may proceed to change locations in order to find the second user's vehicle as described by the messaging data in FIG. 6a and/or the image provided in FIG. 6b. As the end user views the second user's vehicle through the smart device or other augmented reality capable device, the situation-specific tag 603 appears on the display 818 as shown in FIG. 6d, providing a tagged description and/or indicator identifying the vehicle so the end user knows which vehicle matches the one identified within the context of the conversation between the end user and the second user, based on vehicle description, image of the vehicle and/or additional streaming inputs entering through the personal medical channel from one or more additional data sources 119.”);and
in accordance with a determination that the current FOV of the physical environment includes the real-world object corresponding to the text portion of the electronic message, presenting to the user, via the display device, an extended reality (XR) object including the electronic message in association with the real-world object (see at least t[0059] as shown in Fig.6c-6d “Referring to the drawings, FIGS. 6c-6d provide an example of an embodiment of vision assistance that may be outputted by the medical services 117 to an end user device 101. In the example of FIG. 6c-6d, IoT devices, such as a smart display, may utilize augmented reality, vision recognition and other imaging techniques to provide descriptive overlays comprising situation-specific tags 603 onto the smart display. The overlays may be positioned onto or adjacent to physical objects viewed through the display 818 based on where the end user's gaze is directed. The one or more physical objects within the field of vision of the smart display or other types of human-readable displays 818 may be tagged with a situational-specific tag as shown in FIG. 6d. In the example of FIG. 6c-6d, can be a continuation of the example described above for FIG. 6a-6b. Continuing from the FIG. 6a-6b example, an end user equipped with a smart display or another computing device for viewing one or more physical objects, may proceed to change locations in order to find the second user's vehicle as described by the messaging data in FIG. 6a and/or the image provided in FIG. 6b. As the end user views the second user's vehicle through the smart device or other augmented reality capable device, the situation-specific tag 603 appears on the display 818 as shown in FIG. 6d, providing a tagged description and/or indicator identifying the vehicle so the end user knows which vehicle matches the one identified within the context of the conversation between the end user and the second user, based on vehicle description, image of the vehicle and/or additional streaming inputs entering through the personal medical channel from one or more additional data sources 119.”) Trim is understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Goodrich teaches obtaining an electronic message addressed to a user of the computing system in a physical environment while displaying a current field-of-view (FOV) of a physical environment on the display device(see at least [0123] as shown in Fig. 7 “ For example, the screen 700 shows a view of a first portion of a real-world environment 701 through the lenses 512 of the eyewear device 119. The eyewear device 119 may receive input from the user that requests to view a conversation interface. In response, the eyewear device 119 obtains from the client device 102 and/or from the application servers 114 a conversation interface 702 that includes one or more virtual conversation objects. The virtual conversation objects include chat bubbles 703 and 705? representing individual messages sent by respective participants in the conversation and a message generation option 704 (e.g., a microphone) allowing the user to generate a new message for transmission to the participants in the conversation interface. The eyewear device 119 displays the virtual objects 702 as three-dimensional objects in a center of the lenses 512. This makes it appear to the user viewing the first portion of the real-world environment 701 through the lenses 512 as if the virtual objects 702 are within the first portion of the real-world environment 701.”); wherein the electronic message is sent from a sender to the user in a messaging application and includes a text portion indicated ([0037] Image processing server 122 is used to implement scan functionality of the augmentation system 208. Scan functionality includes activating and providing one or more augmented reality experiences on a client device 102 when an image is captured by the client device 102. Specifically, the messaging application 104 on the client device 102 can be used to activate a camera. The camera displays one or more real-time images or a video to a user along with one or more icons or identifiers of one or more augmented reality experiences. The user can select a given one of the identifiers to launch the corresponding augmented reality experience. Launching the augmented reality experience includes obtaining one or more augmented reality items associated with the augmented reality experience and overlaying the augmented reality items on top of the images or video being presented. [0089] message text payload 404: text, to be generated by a user via a user interface of the client device 102, and that is included in the message 400.”; [0122] FIGS. 7-9 are illustrative screens of a graphical user interface of the conversation interface system 107 according to example embodiments. The screens shown in FIGS. 7-9 may be provided by the messaging client 104 of one or more client devices 102, other applications implemented on one or more client devices 102, or the eyewear device 119.” [0123] For example, the screen 700 shows a view of a first portion of a real-world environment 701 through the lenses 512 of the eyewear device 119. The eyewear device 119 may receive input from the user that requests to view a conversation interface. In response, the eyewear device 119 obtains from the client device 102 and/or from the application servers 114 a conversation interface 702 that includes one or more virtual conversation objects. The virtual conversation objects include chat bubbles 703 and 705? representing individual messages sent by respective participants in the conversation and a message generation option 704 (e.g., a microphone) allowing the user to generate a new message for transmission to the participants in the conversation interface. The eyewear device 119 displays the virtual objects 702 as three-dimensional objects in a center of the lenses 512. This makes it appear to the user viewing the first portion of the real-world environment 701 through the lenses 512 as if the virtual objects 702 are within the first portion of the real-world environment 701), via an attachment flag or metadata provided during composition of the electronic message, by the sender as associated with object ([0087] FIG. 4 is a schematic diagram illustrating a structure of a message 400, according to some examples, generated by a messaging client 104 for communication to a further messaging client 104 or the messaging server 118. The content of a particular message 400 is used to populate the message table 302 stored within the database 126, accessible by the messaging server 118. Similarly, the content of a message 400 is stored in memory as “in-transit” or “in-flight” data of the client device 102 or the application servers 114. A message 400 is shown to include the following example components:”…[0097] message tag 420: each message 400 may be tagged with multiple tags, each of which is indicative of the subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value may be included within the message tag 420 that is indicative of the relevant animal. Tag values may be generated manually, based on user input, or may be automatically generated using, for example, image recognition.” where tag is considered as metadata)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of overlaying the specific tag onto the image based on text message description of Trim with obtaining an electronic message including tags during composition of the electronic message while displaying a current field of view of a physical environment on the display device as seen in Goodrich because this modification would display multiple types of virtual objects together within view of a real word environment ([0028] of Goodrich). Both Trim and Goodrich are understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Tiutiunnik teaches a text portion indicated, via an attachment flag or metadata provided during composition of the electronic message, by the sender as associated with a real- world object (see at least Fig. 7, col. 15, lines 64-67-col.16, lines 1-60 “ For example, at least one of the Social Media users (101) using the user device (109), in particular using the (mobile) application, associates (adds) a message he created 106 with an object of interest, in particular the real-world object 104, mentioned association (adding) being performed on the user device (109). It should be noted that a user may (using the camera of the user mobile device 109, or by other functionally similar means), for example, before directing, after directing, or when directing the camera of the user device (109), add content (for example, message text, image, video, audio, hyperlink, etc.), which can be added to (associated with) an object of interest, or in addition to at least one message added by at least one different user. In a particular case, the user can add content using search for an augmented reality tag, etc., to which new content will be added using the user device (109), mentioned tags can be found by criteria in a sorted list, for example, sorted by the distance to the augmented reality tags (in particular to real-world objects), by hashtags associated with the tag, by the date of creation of tags, etc. It should be noted that, after selecting content, including a message, comment, etc., which will be associated with (added to) an object of interest, etc., the user can select an object of interest using the camera of his or her device, including focusing the camera of his or her user device on such an object. After confirming the addition of content, such as a message, etc. to an object of interest (in particular a real-world object), at least data, in particular metadata (for) such an object is transmitted, in particular is loaded into a real-world object identities storage 710 that stores such data and at least object identifiers (for real-world objects, objects of interest). Mentioned data, in particular metadata, is data that can be used to identify (with a predetermined identification reliability level) the tags of augmented reality and/or an object using the camera of the user device, in particular in an image created using the camera of the user device and, for example, stored in the user device memory.……. It should be noted that at least one Social Media user (101, 706) can view mentioned content (in particular messages, etc.) added to at least one of mentioned objects by at least one user. It should be noted that the user 706 can interact with the message, for example, in order to interact with another user 101 of the same Social Media, in particular with the user who added the message, or with other users who interacted therewith, for example, viewed, commented, added content, etc., to such a content, and/or at least with one of the users who was present and/or will be present in the area of approach (for example, 602, 604, FIG. 6) and/or nearby (in particular in close proximity to) the object of interest, in particular the real-world object 104, to which at least one message was added”);
in response to determining that the electronic message indicated by the sender as associated with the real-world object (see at least Fig. 7, col. 15, lines 64-67-col.16, lines 1-60 “ For example, at least one of the Social Media users (101) using the user device (109), in particular using the (mobile) application, associates (adds) a message he created 106 with an object of interest, in particular the real-world object 104, mentioned association (adding) being performed on the user device (109). It should be noted that a user may (using the camera of the user mobile device 109, or by other functionally similar means), for example, before directing, after directing, or when directing the camera of the user device (109), add content (for example, message text, image, video, audio, hyperlink, etc.), which can be added to (associated with) an object of interest, or in addition to at least one message added by at least one different user. In a particular case, the user can add content using search for an augmented reality tag, etc., to which new content will be added using the user device (109), mentioned tags can be found by criteria in a sorted list, for example, sorted by the distance to the augmented reality tags (in particular to real-world objects), by hashtags associated with the tag, by the date of creation of tags, etc. It should be noted that, after selecting content, including a message, comment, etc., which will be associated with (added to) an object of interest, etc., the user can select an object of interest using the camera of his or her device, including focusing the camera of his or her user device on such an object. After confirming the addition of content, such as a message, etc. to an object of interest (in particular a real-world object), at least data, in particular metadata (for) such an object is transmitted, in particular is loaded into a real-world object identities storage 710 that stores such data and at least object identifiers (for real-world objects, objects of interest). Mentioned data, in particular metadata, is data that can be used to identify (with a predetermined identification reliability level) the tags of augmented reality and/or an object using the camera of the user device, in particular in an image created using the camera of the user device and, for example, stored in the user device memory.……. It should be noted that at least one Social Media user (101, 706) can view mentioned content (in particular messages, etc.) added to at least one of mentioned objects by at least one user. It should be noted that the user 706 can interact with the message, for example, in order to interact with another user 101 of the same Social Media, in particular with the user who added the message, or with other users who interacted therewith, for example, viewed, commented, added content, etc., to such a content, and/or at least with one of the users who was present and/or will be present in the area of approach (for example, 602, 604, FIG. 6) and/or nearby (in particular in close proximity to) the object of interest, in particular the real-world object 104, to which at least one message was added”), determining whether the current FOV of the physical environment includes the real-world object corresponding to the electronic message (at least Fig. 7, col. 15, lines 64-67-col.16, lines 1-60 “ For example, at least one of the Social Media users (101) using the user device (109), in particular using the (mobile) application, associates (adds) a message he created 106 with an object of interest, in particular the real-world object 104, mentioned association (adding) being performed on the user device (109). It should be noted that a user may (using the camera of the user mobile device 109, or by other functionally similar means), for example, before directing, after directing, or when directing the camera of the user device (109), add content (for example, message text, image, video, audio, hyperlink, etc.), which can be added to (associated with) an object of interest, or in addition to at least one message added by at least one different user. In a particular case, the user can add content using search for an augmented reality tag, etc., to which new content will be added using the user device (109), mentioned tags can be found by criteria in a sorted list, for example, sorted by the distance to the augmented reality tags (in particular to real-world objects), by hashtags associated with the tag, by the date of creation of tags, etc. It should be noted that, after selecting content, including a message, comment, etc., which will be associated with (added to) an object of interest, etc., the user can select an object of interest using the camera of his or her device, including focusing the camera of his or her user device on such an object. After confirming the addition of content, such as a message, etc. to an object of interest (in particular a real-world object), at least data, in particular metadata (for) such an object is transmitted, in particular is loaded into a real-world object identities storage 710 that stores such data and at least object identifiers (for real-world objects, objects of interest). Mentioned data, in particular metadata, is data that can be used to identify (with a predetermined identification reliability level) the tags of augmented reality and/or an object using the camera of the user device, in particular in an image created using the camera of the user device and, for example, stored in the user device memory.”, col.16, lines 66-67-col.17, lines 1-34 “The mobile application installed on the user device (707) displays on the user device the message 106 attached to the real-world object 104. For example, the mobile application receives the geographic coordinates of the user device 707 and the user ID. The coordinates of the user device are used by the mobile application to determine the proximity zone (712). The proximity zone 712 is used for pre-filtering, by the user device, messages attached to real-world objects, wherein only those messages are selected which are within possible direct vision range or/and in close proximity to the user 706. Messages selected in this fashion are displayed to the user 706 by the mobile application of the user device 707, in case the mobile application recognizes the augmented reality tags to which mentioned messages were attached. If the user 706 gets within close proximity the real-world object 104 (or within object itself), the message 106 that was added to this real-world object by the another Social Media user 101 will be selected as one of the messages to be displayed provided that the user device 706 recognizes it using the mobile application installed on mentioned user device.. ..The real-world objects 714 captured by the user device are compared with augmented reality tags that are being stored in a real-world object identifies storage 710, mentioned tags being associated with (added to) real-world objects that are within a direct vision or /and in close proximity to the user 706.” Where after confirming the addition of content, such as a message, etc. to an object of interest (in particular a real-world object), at least data, in particular metadata (for) such an object is transmitted, in particular is loaded into a real-world object identities storage 710 that stores such data and at least object identifiers (for real-world objects, objects of interest) then the real-world objects 714 captured by the user device are compared with augmented reality tags that are being stored in a real-world object identifies storage 710 which is considered comparing the real-world objects captured with tags from the message and within a direct vision and in close proximity to the user is considered as current FOV; Col.19. lines 29-67-col.20. lines 1-67,col.21, lines 1-45 “As a part of a process described herein, an artificial neural network (including a third-party general-purpose object recognition artificial neural network that had been taught to recognize a limited number of object categories) 806 is being utilized for object recognition (by the user device and/or server). The input of such neural network is supplied with images (804, 805) received from the camera of the device 109. FIG. 8 shows an image 802 of a real-world object (for example, 104, FIG. 1) to which the Social Media user (101, FIG. 1) can add (associate with) a social media content in form of, for example, a message. Every video frame “slice” can be processed by general purpose image classification Artificial Neural Network. The user device receives an image from the user device memory using a specialized algorithm (in particular the algorithm of conversion of a raster image used by the operating system (for example, RGB or YUV format) into a triplet color component array (RGB—red, green, blue) expressed in real numbers and used by the neural network). For example, obtaining at least one image of an object can be implemented by way of scaling of such an image obtained from the memory of the user device, in particular by enlarging the image 803. In the process of using the artificial neural network, the neural network provides a set of (abstract) attributes of an object 809 that are being generated from real-world object similarities with a number of objects from a list of objects that this given artificial neural network had been taught to recognize. Such a set of abstract attributes (corresponding to those utilized by inner classification process of an artificial neural network that is being used) is uploaded into the real-world object identities storage 710, where each object (812, 813) corresponds to an array of sets of attributes (taking into account different levels of scaling, different angles, lighting, object recognition steps, etc.). Similarities (visual associations) with multiple objects classifiable by Artificial said Neural Network are prioritized by confidence level and stored, creating a “Real World Identity Atom”. To recognize the object stored in the database, the reverse process is being applied, since the set of attributes 810 is obtained by an artificial neural network 808 when processing an image that contains a real-world object that is a recognition candidate 807. The resulting set of attributes 810 is compared with a set of attributes obtained earlier 809 and stored in the real-world object identities storage 710. At the viewer's side, a “Real World Identity Atom” is formed and compared with multitude of stored Atoms, matching similarities allowing to identify a marked object to display a message attached to it. The results of this comparison are being used by the software (for example, a mobile application), installed on the user device, in such a way that mentioned mobile application can decide whether the object with which a message was associated with (attached to) is present or not present within the field of view of the camera of the user device. The mechanism of recognition of real-world objects includes detection of real-world objects in an image which uses the (specialized) ANNRecognizer program module…..);
and in accordance with a determination that the current FOV of the physical environment includes the real-world object, presenting to the user, via the display device, an extended reality (XR) object that corresponds to the electronic message in association with the real-world object (col.14, lines 35-67, col.15-col.19,lines 1-25 as shown in Fig. 6 and Fig.7; col. 17, lines 29-42 “The real-world objects 714 captured by the user device are compared with augmented reality tags that are being stored in a real-world object identities storage 710, mentioned tags being associated with (added to) real-world objects that are within a direct vision or/and in close proximity to the user 706. FIG. 8 illustrates the procedure for such comparison. The real-world object 714, which was recognized as a result of the comparison, can be outlined by a frame 716 or other visual means. The user 706 can view the message 106 displayed on the screen of the user device 717, interact therewith and/or interact with other users who created it, interacted therewith or will interact therewith in the future, using a user interface of software installed on the user device 717”; col.19. ; lines 11-21 “The data received (including metadata) by the user device from the server is compared by the user device with the processed data from the camera of the user device, and the user device thereby determines the visibility of a certain object 714. Mentioned object is then marked for user as being found in any fashion, and content is then added thereto in augmented reality (715, 717). Thus, the added content can be outlining with a frame, placing, in the augmented reality, visual content including text content, over or on the object, or other visual effects”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of overlaying the specific tag onto the image based on text message description of Trim and Goodrich with generating content such as text messages in association with real world objects of Tiutiunnik because this modification would share such content with different users, in particular with at least one community (col.14, lines 25-35 of Tiutiunnik).
In the same field of endeavor, Grant teaches wherein the electronic message is sent from a sender to the user and includes a text portion indicated, via an attachment flag or metadata provided during composition of the electronic message, by the sender as associated with a real- world object (see at least [0094] With this brief illustration of the use case, turning back to FIG. 4, and referring to FIG. 5, which is an example process 500 that may be illustrated by the use case, Jordan puts on her AR glasses as the TU device 420 before heading off to work, walks into the geofenced living room of their apartment (operation 505) and, with her TU device 420 in a tagging mode of operation, scans the living room for anything that needs to be done that day, since she plans on working late. Each room in the apartment may be bounded by a geofence that has been established by Zach and Jordan or by some other entity. A precise location of Jordan's TU device 420 may be established by a location processor 430. The location processor 430 may determine the precise location using GPS, Wi-Fi, GNSS, cellular telephone signals, proximity sensors, image recognition, and the like. Thus, the location processor 430 may determine not only that Jordan is standing in the living room of their apartment, but where within the room she is located. The location processor 430 may further determine the orientation of Jordan's head based on input received from a sensor/imaging unit 422, and may further determine a direction that her eyes are focused in by imaging her pupil location relative to her eye sockets or head. The sensor/imaging unit 422 may be any type of camera or combination of cameras, sensors, and other mechanisms for obtaining an electronic image of the scene 600 being viewed. [0095] As Jordan scans the living room for things that need to be done, she spots the plant 610 and remembers that she forgot to water it. Since she is running late for work, she decides to leave Zach a note to water the plant. In operation 510, Jordan focuses on the object, the plant 610, and selects it. The selecting of the object may be done in a number of ways, and may be assisted using an object determiner 424. Selection of an object may be done by the user in a number of ways. For example, staring at the object for some predetermined amount of time, e.g., two seconds, might signal to the TU device 420 that this object is to be selected. The object determiner 424 may combine information from the sensor/imaging unit 422 with information from the location processor 430, such as precise position, head orientation, and line of sight orientation, to determine specifically the object that is being focused upon. In some embodiments, the TU 410 may speak, and voice recognition and/or speech-to-text translation may be utilized so that the object may be recognized with more precision. For example, Jordan may look at the plant 610 and say “flower pot” or “plant” to assist the object determiner 424. This may help to distinguish between the plant as the object vs. the table with the plant as the object. Various mechanisms may be utilized for leaving the message, including using a drawing motion, writing, typing, speaking, etc. The information associated/bound with the object may include a text message, audio, voice and/or video information provided by the TU 410. [0099] In operation 515, the TU 410 may provide information that allows the message attachment processor 432 to attach a message or any other information as tagging information, including the detected or obtained attribute information to that object. As can be seen in FIGS. 6A and 6B illustrating the use case, the attached information 620 may include the user message (“Can you please water this plant?”) along with location information (e.g., absolute location information related to the geofenced area, and relative location information related to the object's position with respect to other objects in the area) and attributes of the object (object name, size, descriptors, and the like). Although FIG. 5 shows operation 520 following operation 515, the order of these operations may be reversed as well. The information from the TU 410, in some embodiments, may be entered by the TU 410 speaking and the spoken message being translated into text by a speech-to-text translator. In some embodiments, the user may type the message in using their TU device 420. An actual image of the object may be collected and associated with the information that is stored with the object and presented later to the RU 450.”) wherein the association of the text portion with the real-world object is generated in response to a selection input by the sender directed to the real-world object during composition of the electronic message (see at least [0091] FIG. 4 is a block diagram illustrating various components of an AR system 400. A tagging user (TU) 410 is a user who tags various real-world objects within a given location with information that may be available for a subsequent reviewing user (RU) 450 who enters the location at a subsequent time. Each user 410, 450 may have their own respective AR device—the TU device 420 for the TU 410 and the RU device 420′ for the RU 450. The AR devices 420, 420′ may be devices such as glasses, a smart phone, a wearable, or any entity with a camera and/or transparent overlay capable of display functions. Both the TU 410 and the RU 450, as described herein, may view the real-world objects via an AR view from the TU device 420 or the RU device 420′. [0094-0095], [0099] In operation 515, the TU 410 may provide information that allows the message attachment processor 432 to attach a message or any other information as tagging information, including the detected or obtained attribute information to that object. As can be seen in FIGS. 6A and 6B illustrating the use case, the attached information 620 may include the user message (“Can you please water this plant?”) along with location information (e.g., absolute location information related to the geofenced area, and relative location information related to the object's position with respect to other objects in the area) and attributes of the object (object name, size, descriptors, and the like). Although FIG. 5 shows operation 520 following operation 515, the order of these operations may be reversed as well. The information from the TU 410, in some embodiments, may be entered by the TU 410 speaking and the spoken message being translated into text by a speech-to-text translator. In some embodiments, the user may type the message in using their TU device 420. An actual image of the object may be collected and associated with the information that is stored with the object and presented later to the RU 450. [0100] The information collected that is related to the object may be stored in a tagged object database (TODB) 482 that may be located, e.g., on a server 480. The server 480 may be a local server located near the object or in a geofenced region of the object. In this case, IoT technology may be utilized in the collection of and access to the object. The server 480 may also be a remote server and make use of, e.g., cloud technology for the storage of tagged object information. FIG. 5 shows the TODB 482 grouped as a geofenced room with object tag associations 525. The information items stored in the TODB 482 may be provided with access control so only certain users are permitted to see them. For example, the message to water the plant may be tagged by Jordan so that only Zach can see it. In some embodiments, access control may be provided for the individual items of data associated with an object. For example, Jordan may tag the plant 610 with a message to water it, but may simply make a personal note to herself in a tag that she should purchase additional fertilizer for the plant 610—in this way, Zach may be only presented with Jordan's additional information that she intends for him to see”),
in response to determining that the electronic message includes the text portion indicated by the sender as associated with the real-world object, determining whether the current FOV of the physical environment includes the real-world object corresponding to the text portion of the electronic message ([0102] Turning to operation 530, an RU 450 enters an area, such as a geofenced area, containing objects that have been tagged. In the use case, this may be Zach entering the living room in which the plant 610 was tagged by Jordan. The RU device 420′ may contain similar elements to the TU device 420 described above, and the description of these elements is not repeated here. In this situation, the RU device 420′ may be in a reviewing mode of operation. The location processor 430′ of the RU device 420′ may detect entry into the geofenced area of the living room, in the use case, and the RU device 420′ may access the TODB 482 on the server 480 to obtain information on tagged objects within the geofenced area.”; [0104] In operation 535, the RU device 420′ may be loaded with objects and respective tags contained within the geofenced area from the TODB 482, based on a detected position of the RU device 420′, as detected by the location processor 430′. Next, in operation 540, the RU device 420′ may scan for objects within the geofenced area using its sensor/imaging unit 422′. The object determiner 424′ and object attribute determiner 426′ may operate as described above with respect to the TU device. Tagged objects within the TODB 482 may be located and the attributes of each object may be obtained by the RU device 420′. In the use case, the imaging unit 422′ identifies the plant 610 as illustrated in FIG. 6B. As noted previously, however, the plant 610 has been moved on the table and rotated by the cleaning person so that only a part of the graphic 612 is seen. Nonetheless, the RU device 420′ is able to make a positive object determination using the object determiner 424′ and the object attribute determiner 426′ to identify the object as stored in the TODB 482.”); and in accordance with a determination that the current FOV of the physical environment includes the real-world object corresponding to the text portion of the electronic message, presenting to the user, via the display device, an extended reality (XR) object including the electronic message in association with the real-world object ([0105] In some implementations, a “match” of the detected object by the RU device 420′ may be made according to some predefined degree of certainty or a predefined matching criteria. Known matching algorithms may be implemented that perform a determination of a degree of matching between the object (or its descriptive/captured attributes) sensed by the RU device 420′ and the object (or its descriptive/captured attributes) for which the attributes and message are attached. In the use case, even though the graphic 612 was rotated, and cannot be completely seen in FIG. 6B, when compared with the scene 600 in FIG. 6A, a match may be made, e.g., with a 95% certainty, which may be above a predefined threshold of 90%, and thus constitute a match for the object that was the subject of the message left by the TU 410. Any of the determined attributes may be brought into the analysis to determine a matching certainty. An image of the object stored in the TODB 482 may be presented to the RU 450. In some embodiments, such an image may be presented every time the object determiner 424′ determines an object, and in some embodiments, such an image may only be presented when there is some threshold level of uncertainty associated with the object recognition. [0106] With the match by the RU device 420′ of the particular object corresponding to an object stored in the TODB 482, in operation 545, the RU 450 may be presented with a text or graphic overlay of the message and possibly other information associated with the object (in the use case, the plant 610) by a message receiver 434′. This other information may include some or all of the additional information associated with the object. The RU 450 may switch roles with the TU 410 and add their own tagged information related to the object. In the use case, Zach may go ahead and water the plant 610, and then leave a message for Jordan using the procedure described above indicating, e.g., “I added two cups of water to the flowers. Is that enough?”. One or more of the original tags associated by the object may be updated or removed, as described above.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of overlaying the specific tag onto the image based on text message description of Trim, Goodrich and generating content such as text messages in association with real world objects of Tiutiunnik with selecting of a real-world object from a tagging user who views the object as tagging information as seen in Grant because this modification would attach a message or any other information as tagging information, including the detected or obtained attribute information to that object ([0099] of Grant)
Thus, the combination of Trim, Goodrich, Tiutiunnik and Grant teaches a method comprising: at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices via a communication interface: obtaining an electronic message addressed to a user of the computing system in a physical environment while displaying a current field-of-view (FOV) of the physical environment on the display device, wherein the electronic message is sent from a sender to the user in a messaging application and includes a text portion indicated, via an attachment flag or metadata provided during composition of the electronic message, by the sender as associated with a real- world object, wherein the association of the text portion with the real-world object is generated in response to a selection input by the sender directed to the real-world object during composition of the electronic message; in response to determining that the electronic message includes the text portion indicated by the sender as associated with the real-world object, determining whether the current FOV of the physical environment includes the real-world object corresponding to the text portion of the electronic message; and in accordance with a determination that the current FOV of the physical environment includes the real-world object corresponding to the text portion of the electronic message, presenting to the user, via the display device, an extended reality (XR) object including the electronic message in association with the real-world object.
Regarding claim 2, Trim, Goodrich, Tiutiunnik and Grant teach the method of claim 1, further comprising: in accordance with a determination that the current FOV does not include the real-world object, forgoing presentation of the XR object in association with the real-world object (see at least [0059] as shown in Fig.6c of Trim“ Referring to the drawings, FIGS. 6c-6d provide an example of an embodiment of vision assistance that may be outputted by the medical services 117 to an end user device 101. In the example of FIG. 6c-6d, IoT devices, such as a smart display, may utilize augmented reality, vision recognition and other imaging techniques to provide descriptive overlays comprising situation-specific tags 603 onto the smart display. The overlays may be positioned onto or adjacent to physical objects viewed through the display 818 based on where the end user's gaze is directed. The one or more physical objects within the field of vision of the smart display or other types of human-readable displays 818 may be tagged with a situational-specific tag as shown in FIG. 6d. In the example of FIG. 6c-6d, can be a continuation of the example described above for FIG. 6a-6b. Continuing from the FIG. 6a-6b example, an end user equipped with a smart display or another computing device for viewing one or more physical objects, may proceed to change locations in order to find the second user's vehicle as described by the messaging data in FIG. 6a and/or the image provided in FIG. 6b. As the end user views the second user's vehicle through the smart device or other augmented reality capable device, the situation-specific tag 603 appears on the display 818 as shown in FIG. 6d, providing a tagged description and/or indicator identifying the vehicle so the end user knows which vehicle matches the one identified within the context of the conversation between the end user and the second user, based on vehicle description, image of the vehicle and/or additional streaming inputs entering through the personal medical channel from one or more additional data sources 119.”).
Regarding claim 3, Trim, Goodrich, Tiutiunnik and Grant teach the method of claim 1, herein the electronic message includes: the metadata indicating a type of the real-world object, and wherein determining whether the current FOV of the physical environment includes the real-world object comprises performing an object classification technique to identify an object matching the type of the real-world object; the metadata indicating a representation of the real-world object, and wherein determining whether the current FOV of the physical environment includes the real-world object comprises performing an object detection technique using the representation of the real-world object; or the metadata indicating a location of the real-world object, and wherein determining whether the current FOV of the physical environment includes the real-world object comprises determining whether an object in the current FOV is at a location corresponding to the location of the real-world object (see at least [0056], [0059] of Trim “[0058] In some embodiments, additional data may be collected from one or more additional data sources 119 to provide more context or environmental information. The additional data can be analyzed by the medical services to further understand the situationally specific data being presented to the end user. For example, the text message presented in FIG. 6a indicates a location of the second user's vehicle as being outside the end user's 601 building. Embodiments of the medical services 117 may receive or retrieve additional inputs from additional data sources 119, such as location services 241, sensor readings, IoT device communications, data and metadata from an end user device 101 or another client device on the communication network, including video streaming data from one or more client devices. For example, referring back to FIG. 6a-6b, additional context to pinpoint the location and position of the vehicle described, relative to the location of the end user device receiving the image of the vehicle being described. For instance, additional data sources 119 could be accessed to pinpoint the vehicle's location, such as comparing GPS location of the end user 601 and/or the second user via location services 241, readings from one or more sensor devices within the vehicle may measure speed, travel distance, direction, etc. while camera systems or other IoT devices of the vehicle may capture images of the vehicle's surrounding environment which can be further identified by the medical services 117 and cross-checked with the text messaging data and/or vehicle image to ensure the medical services 117 accurately labels the correct vehicle in image sent to the end user 601.” ; [0073] In some embodiments, the medical services 117 may comprise a neighboring eye's information monitor 327 to synchronize object detection and improve accuracy of object detection by employing the use of surrounding devices of the communication network 150 to monitor end user device 101 position and relationships to objects identified as part of the streaming inputs as well as the accuracy of gaze detection by the visual stimuli tracker 323. For example, the medical services 117 may improve object detection and accuracy by leveraging nearby camera systems 507, sensor devices and/or IoT devices to further monitor the actions of the end user, by providing multiple sources and angles for tracking the end user rather than just data collection at a single point. For instance, the environment surrounding the end user and gaze of the end user may be tracked by a plurality of camera systems 507 positioned nearby, each recording from a different angle, to provide additional data about surrounding objects, object positions and eye positioning of the end user. For example, employing vision recognition services 301 to detect objects via the camera system 507 connected to the end user device 101 as well as additional vision recognition of objects using one or more cameras, sensors or IoT devices recording the same objects via additional devices connected to the communication network 150 that share the collected data with the medical services 117. [0097] of Goodrich “message tag 420: each message 400 may be tagged with multiple tags, each of which is indicative of the subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value may be included within the message tag 420 that is indicative of the relevant animal. Tag values may be generated manually, based on user input, or may be automatically generated using, for example, image recognition.”;col.16, lines 17-46 of Tiutiunnik “It should be noted that, after selecting content, including a message, comment, etc., which will be associated with (added to) an object of interest, etc., the user can select an object of interest using the camera of his or her device, including focusing the camera of his or her user device on such an object. After confirming the addition of content, such as a message, etc. to an object of interest (in particular a real-world object), at least data, in particular metadata (for) such an object is transmitted, in particular is loaded into a real-world object identities storage 710 that stores such data and at least object identifiers (for real-world objects, objects of interest). Mentioned data, in particular metadata, is data that can be used to identify (with a predetermined identification reliability level) the tags of augmented reality and/or an object using the camera of the user device, in particular in an image created using the camera of the user device and, for example, stored in the user device memory. Such data, in particular metadata, can be arrays of numbers (vectors) obtained using ANN, or tags generated by computer vision algorithms. Furthermore, such mentioned data may be three-dimensional space descriptions obtained using algorithms that process data for display within Augmented Reality, in a particular case, such mentioned data returns software, such as software development kit that allows for augmented reality applications to be built (for example, ARCore®) and/or application programming interface (API) which lets developers build augmented reality applications, for example, with taking advantage of a device's camera, CPU, GPU, motion sensors, etc. (for example, AR Kit) and its format is specified by the libraries themselves”; col. 17, lines 56-67-col.18, lines 1-3 “Pointing the camera of the user device by the user at the object of interest 104, wherein the user mobile application determines not only the visual content, but also attempts to determine the coordinates of the object of interest, in particular with a use of compass direction, meters above sea level, three-dimensional world description (for example, presence of surfaces), the location of a new tag compared with previously added tags (both to the same object of interest and/or the real-world object, and to other the real-world objects and/or objects of interest), for example, using gyroscope or locating, in particular by means of software, for example, augmented reality libraries, in particular provided that other tags were previously recognized using the user device; Col.19, lines 11-21 “The data received (including metadata) by the user device from the server is compared by the user device with the processed data from the camera of the user device, and the user device thereby determines the visibility of a certain object 714. Mentioned object is then marked for user as being found in any fashion, and content is then added thereto in augmented reality (715, 717). Thus, the added content can be outlining with a frame, placing, in the augmented reality, visual content including text content, over or on the object, or other visual effects.” Col.19. lines 29-67-col.20. lines 1-67,col.21, lines 1-45 “As a part of a process described herein, an artificial neural network (including a third-party general-purpose object recognition artificial neural network that had been taught to recognize a limited number of object categories) 806 is being utilized for object recognition (by the user device and/or server). The input of such neural network is supplied with images (804, 805) received from the camera of the device 109. FIG. 8 shows an image 802 of a real-world object (for example, 104, FIG. 1) to which the Social Media user (101, FIG. 1) can add (associate with) a social media content in form of, for example, a message. Every video frame “slice” can be processed by general purpose image classification Artificial Neural Network. The user device receives an image from the user device memory using a specialized algorithm (in particular the algorithm of conversion of a raster image used by the operating system (for example, RGB or YUV format) into a triplet color component array (RGB—red, green, blue) expressed in real numbers and used by the neural network). For example, obtaining at least one image of an object can be implemented by way of scaling of such an image obtained from the memory of the user device, in particular by enlarging the image 803. In the process of using the artificial neural network, the neural network provides a set of (abstract) attributes of an object 809 that are being generated from real-world object similarities with a number of objects from a list of objects that this given artificial neural network had been taught to recognize. Such a set of abstract attributes (corresponding to those utilized by inner classification process of an artificial neural network that is being used) is uploaded into the real-world object identities storage 710, where each object (812, 813) corresponds to an array of sets of attributes (taking into account different levels of scaling, different angles, lighting, object recognition steps, etc.). Similarities (visual associations) with multiple objects classifiable by Artificial said Neural Network are prioritized by confidence level and stored, creating a “Real World Identity Atom”. To recognize the object stored in the database, the reverse process is being applied, since the set of attributes 810 is obtained by an artificial neural network 808 when processing an image that contains a real-world object that is a recognition candidate 807. The resulting set of attributes 810 is compared with a set of attributes obtained earlier 809 and stored in the real-world object identities storage 710. At the viewer's side, a “Real World Identity Atom” is formed and compared with multitude of stored Atoms, matching similarities allowing to identify a marked object to display a message attached to it. The results of this comparison are being used by the software (for example, a mobile application), installed on the user device, in such a way that mentioned mobile application can decide whether the object with which a message was associated with (attached to) is present or not present within the field of view of the camera of the user device. The mechanism of recognition of real-world objects includes detection of real-world objects in an image which uses the (specialized) ANNRecognizer program module…..”; [0095-0097];[0105]-0106] of Grant) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 4, Trim, Goodrich, Tiutiunnik and Grant teach the method of claim 1, further comprising:
obtaining one or more images associated with the current FOV of the physical environment from one or more exterior-facing image sensors associated with the computing system (see at least [0064] of Trim “ In some embodiments of the medical services 117, the medical services 117 may comprise a source stream capture 305 module. Embodiments of the source stream capture 305 module may perform the task or function of the medical services 117 directed toward capturing and storing the input signal of the streaming inputs passing through the personal medical channel 227. The source stream capture 305 module may organize and store the captured input signals of the streaming inputs in one or more data structures which may be allocated as virtual network resources 225. Embodiments of the source stream capture 305 module may include a serial data controller, which may be responsible for performing signal capture and storage on the communications network 150. “[0071] of Goodrich “As described above, augmentation data includes augmented reality content items, overlays, image transformations, AR images, and similar terms that refer to modifications that may be applied to image data (e.g., videos or images). This includes real-time modifications, which modify an image as it is captured using device sensors (e.g., one or multiple cameras) of a client device 102 and then displayed on a screen of the client device 102 with the modifications. This also includes modifications to stored content, such as video clips in a gallery that may be modified. For example, in a client device 102 with access to multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how the different augmented reality content items will modify the stored clip. For example, multiple augmented reality content items that apply different pseudorandom movement models can be applied to the same content by selecting different augmented reality content items for the content. Similarly, real-time video capture may be used with an illustrated modification to show how video images currently being captured by sensors of a client device 102 would modify the captured data. Such data may simply be displayed on the screen and not stored in memory, or the content captured by the device sensors may be recorded and stored in memory with or without the modifications (or both)”; see at least Fig.7, col.16, lines 47-67-col.17, lines 1 of Tiutiunik, “..It should be noted that to view content, for example, messages, the Social Media user (101 or 706) directs the camera of his or her user device (109 or 707) towards mentioned object of interest (real-world object, etc.) or using other methods and means of interaction with the user device (109, 707), for example, by selecting content displayed on the user device. The mobile application installed on the user device (707) displays on the user device the message 106 attached to the real-world object 104.”; col. 17, lines 29-37 “The real-world objects 714 captured by the user device are compared with augmented reality tags that are being stored in a real-world object identities storage 710, mentioned tags being associated with (added to) real-world objects that are within a direct vision or/and in close proximity to the user 706. FIG. 8 illustrates the procedure for such comparison. The real-world object 714, which was recognized as a result of the comparison, can be outlined by a frame 716 or other visual means.”; col.19, lines 66-67-col.20, lines 1-15 “To recognize the object stored in the database, the reverse process is being applied, since the set of attributes 810 is obtained by an artificial neural network 808 when processing an image that contains a real-world object that is a recognition candidate 807. The resulting set of attributes 810 is compared with a set of attributes obtained earlier 809 and stored in the real-world object identities storage 710. At the viewer's side, a “Real World Identity Atom” is formed and compared with multitude of stored Atoms, matching similarities allowing to identify a marked object to display a message attached to it. The results of this comparison are being used by the software (for example, a mobile application), installed on the user device, in such a way that mentioned mobile application can decide whether the object with which a message was associated with (attached to) is present or not present within the field of view of the camera of the user device.”; [0094-0095] of Grant);
obtaining a current physical environment descriptor characterizing the current FOV of the physical environment based on the one or more images(see at least at col. 18, lines 49-67-col.19, lines 1-9 of Tiutiunik “At request from the mobile application, objects that can be “detected” (seen, identified) using the camera of the user device, are uploaded to the user device, in particular to the mobile application, from the server, in particular from the real-world object identities storage 710. For example, metadata is uploaded, in particular tags, real-world objects (and/or objects of interest), based on a proximity zone 712 in which they are located. For example, real-world objects can be uploaded, in particular identifiers thereof, images thereof, content thereof, messages thereof, etc. For example, data related to such real-world objects that are in close proximity to the (geographic) location of the user device, number of nearby objects (for example, 25), including data previously loaded for such objects, or data of such objects (or related to such objects, for example, all objects located not farther than 100 meters from the current location of the user device, or a plurality of such data, in particular criteria, may be uploaded, and, further, data (in particular criteria) of the relevance of objects for such a user may be uploaded to the user device, for example, depending on the user's interests, wherein the tags, being a portion of the data downloaded to the user device, may be associated with each other by social connections of the users with each other, in particular with other users with whom the user is connected, to whose device mentioned data is downloaded, the presence of hashtags, time of tag/message/content creation, etc..”; col. 19, lines 27-65 “ As a part of a process described herein, an artificial neural network (including a third-party general-purpose object recognition artificial neural network that had been taught to recognize a limited number of object categories) 806 is being utilized for object recognition (by the user device and/or server). The input of such neural network is supplied with images (804, 805) received from the camera of the device 109. FIG. 8 shows an image 802 of a real-world object (for example, 104, FIG. 1) to which the Social Media user (101, FIG. 1) can add (associate with) a social media content in form of, for example, a message. Every video frame “slice” can be processed by general purpose image classification Artificial Neural Network. The user device receives an image from the user device memory using a specialized algorithm (in particular the algorithm of conversion of a raster image used by the operating system (for example, RGB or YUV format) into a triplet color component array (RGB—red, green, blue) expressed in real numbers and used by the neural network). For example, obtaining at least one image of an object can be implemented by way of scaling of such an image obtained from the memory of the user device, in particular by enlarging the image 803. In the process of using the artificial neural network, the neural network provides a set of (abstract) attributes of an object 809 that are being generated from real-world object similarities with a number of objects from a list of objects that this given artificial neural network had been taught to recognize. Such a set of abstract attributes (corresponding to those utilized by inner classification process of an artificial neural network that is being used) is uploaded into the real-world object identities storage 710, where each object (812, 813) corresponds to an array of sets of attributes (taking into account different levels of scaling, different angles, lighting, object recognition steps, etc.). Similarities (visual associations) with multiple objects classifiable by Artificial said Neural Network are prioritized by confidence level and stored, creating a “Real World Identity Atom”; [0094-0097] of Grant); and wherein determining whether the current FOV of the physical environment includes the real-world object includes determining whether the current physical environment descriptor characterizing the current FOV of the physical environment includes information associated with the real-world object. (see at least [0059] of Trim “Referring to the drawings, FIGS. 6c-6d provide an example of an embodiment of vision assistance that may be outputted by the medical services 117 to an end user device 101. In the example of FIG. 6c-6d, IoT devices, such as a smart display, may utilize augmented reality, vision recognition and other imaging techniques to provide descriptive overlays comprising situation-specific tags 603 onto the smart display. The overlays may be positioned onto or adjacent to physical objects viewed through the display 818 based on where the end user's gaze is directed. The one or more physical objects within the field of vision of the smart display or other types of human-readable displays 818 may be tagged with a situational-specific tag as shown in FIG. 6d. In the example of FIG. 6c-6d, can be a continuation of the example described above for FIG. 6a-6b. Continuing from the FIG. 6a-6b example, an end user equipped with a smart display or another computing device for viewing one or more physical objects, may proceed to change locations in order to find the second user's vehicle as described by the messaging data in FIG. 6a and/or the image provided in FIG. 6b. As the end user views the second user's vehicle through the smart device or other augmented reality capable device, the situation-specific tag 603 appears on the display 818 as shown in FIG. 6d, providing a tagged description and/or indicator identifying the vehicle so the end user knows which vehicle matches the one identified within the context of the conversation between the end user and the second user, based on vehicle description, image of the vehicle and/or additional streaming inputs entering through the personal medical channel from one or more additional data sources 119”; see at least at col.19, lines 11-21 of Tiutiunik “The data received (including metadata) by the user device from the server is compared by the user device with the processed data from the camera of the user device, and the user device thereby determines the visibility of a certain object 714. Mentioned object is then marked for user as being found in any fashion, and content is then added thereto in augmented reality (715, 717). Thus, the added content can be outlining with a frame, placing, in the augmented reality, visual content including text content, over or on the object, or other visual effects”; col.19, lines 66-67-col.20, lines 1-15 “To recognize the object stored in the database, the reverse process is being applied, since the set of attributes 810 is obtained by an artificial neural network 808 when processing an image that contains a real-world object that is a recognition candidate 807. The resulting set of attributes 810 is compared with a set of attributes obtained earlier 809 and stored in the real-world object identities storage 710. At the viewer's side, a “Real World Identity Atom” is formed and compared with multitude of stored Atoms, matching similarities allowing to identify a marked object to display a message attached to it. The results of this comparison are being used by the software (for example, a mobile application), installed on the user device, in such a way that mentioned mobile application can decide whether the object with which a message was associated with (attached to) is present or not present within the field of view of the camera of the user device” ; [0105-0106] of Grant) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 6, Trim, Goodrich, Tiutiunnik and Grant teach the method of claim 1, wherein presenting the XR object in association with the real-world object includes one of presenting the XR object overlaid on the real-world object or presenting the XR object adjacent to the real-world object (see at least [0059] of Trim “Referring to the drawings, FIGS. 6c-6d provide an example of an embodiment of vision assistance that may be outputted by the medical services 117 to an end user device 101. In the example of FIG. 6c-6d, IoT devices, such as a smart display, may utilize augmented reality, vision recognition and other imaging techniques to provide descriptive overlays comprising situation-specific tags 603 onto the smart display. The overlays may be positioned onto or adjacent to physical objects viewed through the display 818 based on where the end user's gaze is directed. The one or more physical objects within the field of vision of the smart display or other types of human-readable displays 818 may be tagged with a situational-specific tag as shown in FIG. 6d. In the example of FIG. 6c-6d, can be a continuation of the example described above for FIG. 6a-6b. Continuing from the FIG. 6a-6b example, an end user equipped with a smart display or another computing device for viewing one or more physical objects, may proceed to change locations in order to find the second user's vehicle as described by the messaging data in FIG. 6a and/or the image provided in FIG. 6b. As the end user views the second user's vehicle through the smart device or other augmented reality capable device, the situation-specific tag 603 appears on the display 818 as shown in FIG. 6d, providing a tagged description and/or indicator identifying the vehicle so the end user knows which vehicle matches the one identified within the context of the conversation between the end user and the second user, based on vehicle description, image of the vehicle and/or additional streaming inputs entering through the personal medical channel from one or more additional data sources 119.”; Figs 7-8 of Goodrich; see at least Figs 6-7,col.17, lines 29-42 of Tiutiunik “The real-world objects 714 captured by the user device are compared with augmented reality tags that are being stored in a real-world object identities storage 710, mentioned tags being associated with (added to) real-world objects that are within a direct vision or/and in close proximity to the user 706. FIG. 8 illustrates the procedure for such comparison. The real-world object 714, which was recognized as a result of the comparison, can be outlined by a frame 716 or other visual means. The user 706 can view the message 106 displayed on the screen of the user device 717, interact therewith and/or interact with other users who created it, interacted therewith or will interact therewith in the future, using a user interface of software installed on the user device 717” where Fig. 6 showing message adjacent the real world object; Fig.7 showing the message 106 displayed on the screen of the user device 171 adjacent the real world object 714; col.19, lines 11-21 “The data received (including metadata) by the user device from the server is compared by the user device with the processed data from the camera of the user device, and the user device thereby determines the visibility of a certain object 714. Mentioned object is then marked for user as being found in any fashion, and content is then added thereto in augmented reality (715, 717). Thus, the added content can be outlining with a frame, placing, in the augmented reality, visual content including text content, over or on the object, or other visual effects”; Fig. 6A-6B [ 0106] of Grant “With the match by the RU device 420′ of the particular object corresponding to an object stored in the TODB 482, in operation 545, the RU 450 may be presented with a text or graphic overlay of the message and possibly other information associated with the object (in the use case, the plant 610) by a message receiver 434′. This other information may include some or all of the additional information associated with the object. The RU 450 may switch roles with the TU 410 and add their own tagged information related to the object. In the use case, Zach may go ahead and water the plant 610, and then leave a message for Jordan using the procedure described above indicating, e.g., “I added two cups of water to the flowers. Is that enough?”. One or more of the original tags associated by the object may be updated or removed, as described above.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 7, Trim, Goodrich, Tiutiunnik and Grant teach the method of claim 1, further comprising: in response to obtaining the electronic message, presenting, via the display device, a two-dimensional (2D) representation of the electronic message (see at least [0056-0059] as shown in Fig. 6a-6d of Trim; Figs 7-8 of Goodrich; see at least at col.11, lines 12-28 of Tiutiunik “A message area 203, comprising messages associated with the real-world object 104, in particular added (by Social Media users), contains message text 204, in particular a text or image of at least one message. In a particular case, the message area 203 may be located above the object or on the object itself, or in other way visually related to the object. The message area 203 may contain an avatar of a Social Media user 205, for example, mentioned avatar may be an image, in particular a photo, a picture, video, etc., for example, a user photo (portrait, etc.), and may also be a symbolic avatar or an emoji-avatar. It should be noted that the avatar can be a graphical and/or textual presentation of the user, user's alter ego, game character, internet character, application user (for example, the user of the application 102), etc., wherein the avatar can be a two-dimensional (static or animated) image (for example, an icon) and/or a three-dimensional model, and/or text.”; col.11, lines 62-67-col.12, lines 27 “) In addition, the same motivation is used as the rejection for claim 1.
Regarding independent claim 9, Trim teaches a device comprising: one or more processors; a non-transitory memory ([0089] Computer system 800 may include communications fabric 802, which provides for communications between one or more processors 803, memory 805, persistent storage 806, communications unit 811, and one or more input/output (I/O) interfaces 815. Communications fabric 802 can be implemented with any architecture designed for passing data and/or controlling information between processors 803 (such as microprocessors, communications and network processors, etc.), memory 805, external devices 817, and any other hardware components within a system. For example, communications fabric 802 can be implemented with one or more buses”);
an interface for communicating with a display device and one or more input devices ([0092] Communications unit 811 provides for communications with other computer systems or devices via a network (e.g., communication network 150). In the exemplary embodiment, communications unit 811 may include network adapters or interfaces such as a TCP/IP adapter cards, wireless Wi-Fi interface cards, 3G, 4G, or 5G wireless interface cards or other wired or wireless communication links. The communication network 150 can comprise, for example, copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, edge servers and/or other infrastructure and resources 209 which may be part of the radio nodes 211, cloud nodes 213 and network nodes 215 which create the communication network 150. Software and data used to practice embodiments of the present invention can be downloaded to each of the computer systems operating in computing environment 100 or computer system 800 through communications unit 811 (e.g., via the Internet, a local area network or other wide area network). From communications unit 811, the software and data can be loaded onto persistent storage 806”; [0093] One or more I/O interfaces 815 may allow for input and output of data with other devices that may be connected to computer system 800. For example, I/O interface 815 can provide a connection to one or more external devices 817 such as one or more internet-of-things devices, recording devices such as an audio system, camera systems, one or more sensor device(s), input devices such as a keyboard, computer mouse, touch screen, virtual keyboard, touch pad, pointing device, or other human interface devices. External devices 817 can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. I/O interface 815 may connect to human-readable display 818. Human-readable display 818 provides a mechanism to display data to a user and can be, for example, a computer monitor or screen. Human-readable display 818 can also be an incorporated display and may function as a touch screen, such as a built-in display of a tablet computer.”; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors ([0089] Computer system 800 may include communications fabric 802, which provides for communications between one or more processors 803, memory 805, persistent storage 806, communications unit 811, and one or more input/output (I/O) interfaces 815. Communications fabric 802 can be implemented with any architecture designed for passing data and/or controlling information between processors 803 (such as microprocessors, communications and network processors, etc.), memory 805, external devices 817, and any other hardware components within a system. For example, communications fabric 802 can be implemented with one or more buses.[0090] Memory 805 and persistent storage 806 may be computer-readable storage media. Embodiments of memory 805 may include random access memory (RAM) and cache 807 memory. In general, memory 805 can include any suitable volatile or non-volatile computer-readable storage media. Software applications and services 203, such as medical services 117 may be stored in persistent storage 806 for execution and/or access by one or more of the respective processors 803 of the underlying infrastructure and resources 209 of the communication network 150.”), cause the device to: Remaining limitations of claim 9 is similar scope to claim 1 and therefore rejected under the same rationale.
Regarding claim 10, Trim, Goodrich, Tiutiunnik and Grant teach the device of claim 9, wherein the one or more programs further cause the device to: Remaining limitations of claim 10 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 11, Trim, Goodrich, Tiutiunnik and Grant teach the device of claim 9, wherein the electronic message includes: Remaining limitations of claim 11 is similar scope to claim 3 and therefore rejected under the same rationale.
Regarding claim 12, T Trim, Goodrich, Tiutiunnik and Grant teach the device of claim 9, wherein the one or more programs further cause the device to: Remaining limitations of claim 12 is similar scope to claim 4 and therefore rejected under the same rationale.
Regarding claim 14, Trim, Goodrich, Tiutiunnik and Grant teach the device of claim 9, Remaining limitations of claim 14 is similar scope to claim 6 and therefore rejected under the same rationale.
Regarding independent claim 15, Trim teaches a non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with an interface for communicating with a display device and one or more input devices ([0089] Computer system 800 may include communications fabric 802, which provides for communications between one or more processors 803, memory 805, persistent storage 806, communications unit 811, and one or more input/output (I/O) interfaces 815. Communications fabric 802 can be implemented with any architecture designed for passing data and/or controlling information between processors 803 (such as microprocessors, communications and network processors, etc.), memory 805, external devices 817, and any other hardware components within a system. For example, communications fabric 802 can be implemented with one or more buses.[0090] Memory 805 and persistent storage 806 may be computer-readable storage media. Embodiments of memory 805 may include random access memory (RAM) and cache 807 memory. In general, memory 805 can include any suitable volatile or non-volatile computer-readable storage media. Software applications and services 203, such as medical services 117 may be stored in persistent storage 806 for execution and/or access by one or more of the respective processors 803 of the underlying infrastructure and resources 209 of the communication network 150.” [0092] Communications unit 811 provides for communications with other computer systems or devices via a network (e.g., communication network 150). In the exemplary embodiment, communications unit 811 may include network adapters or interfaces such as a TCP/IP adapter cards, wireless Wi-Fi interface cards, 3G, 4G, or 5G wireless interface cards or other wired or wireless communication links. The communication network 150 can comprise, for example, copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, edge servers and/or other infrastructure and resources 209 which may be part of the radio nodes 211, cloud nodes 213 and network nodes 215 which create the communication network 150. Software and data used to practice embodiments of the present invention can be downloaded to each of the computer systems operating in computing environment 100 or computer system 800 through communications unit 811 (e.g., via the Internet, a local area network or other wide area network). From communications unit 811, the software and data can be loaded onto persistent storage 806. [0093] One or more I/O interfaces 815 may allow for input and output of data with other devices that may be connected to computer system 800. For example, I/O interface 815 can provide a connection to one or more external devices 817 such as one or more internet-of-things devices, recording devices such as an audio system, camera systems, one or more sensor device(s), input devices such as a keyboard, computer mouse, touch screen, virtual keyboard, touch pad, pointing device, or other human interface devices. External devices 817 can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. I/O interface 815 may connect to human-readable display 818. Human-readable display 818 provides a mechanism to display data to a user and can be, for example, a computer monitor or screen. Human-readable display 818 can also be an incorporated display and may function as a touch screen, such as a built-in display of a tablet computer.”), cause the device to: Remaining limitations of claim 15 is similar scope to claim 9 and therefore rejected under the same rationale.
Regarding claim 16, Trim, Goodrich, Tiutiunnik and Grant teach the non-transitory memory of claim 15, wherein the one or more programs further cause the device to: Remaining limitations of claim 16 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 17, Trim, Goodrich, Tiutiunnik and Grant teach the non-transitory memory of claim 15, wherein the electronic message includes: Remaining limitations of claim 17 is similar scope to claim 3 and therefore rejected under the same rationale.
Regarding claim 18, Trim, Goodrich, Tiutiunnik and Grant teach the non-transitory memory of claim 15, wherein the one or more programs further cause the device to: Remaining limitations of claim 18 is similar scope to claim 4 and therefore rejected under the same rationale.
Regarding claim 20, Trim, Goodrich, Tiutiunnik and Grant teach the non-transitory memory of claim 15, Remaining limitations of claim 20 is similar scope to claim 6 and therefore rejected under the same rationale.
2. Claims 5, 13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Trim et al, U.S Patent Application Publication No 2020/0387537 (“Trim”) in view of Goodrich et al, U.S Patent Application Publication No.2022/0197027 (“Goodrich”) further in view of Tiutiunnik et al. U.S Patent No.11430211 (“Tiutiunnik”) further in view of Grant et al., U.S Patent Application Publication No. 20200401802 (“Grant”) further in view of Freese et al., U.S Patent Application No. 2021/0215940 (“Freese”)
Regarding claim 5, Trim, Goodrich, Tiutiunnik and Grant teach the method of claim 1, wherein the XR object corresponds to XR content to the real-world object ( [0056-0059] as shown in Fig. 6a-6d of Trim; [0126] of Goodrich “ As shown in FIG. 8, the eyewear device 119 can display multiple types of virtual objects at the same time. For example, the eyewear device 112 can display a conversation interface 801 and virtual media asset objects 802. The eyewear device 119 can selectively perform a function pertaining to the conversation interface 801 or virtual media asset objects 802 based a distance between a center of the conversation interface 801 and virtual media asset objects 802 and the center 803 of the eyewear device 119. The distance between the centers of the conversation interface 801 and virtual media asset objects 802 and the center 803 of the eyewear device 119 can change as the user physically moves around and looks at different portion of the real-world environment. This is because the conversation interface 801 and virtual media asset objects 802 can remain statically positioned as the user physically moves around to look at different portions of the real-world environment.” see at least Fig. 7, col. 17, lines 29-42 of Tiutiunik “The real-world objects 714 captured by the user device are compared with augmented reality tags that are being stored in a real-world object identities storage 710, mentioned tags being associated with (added to) real-world objects that are within a direct vision or/and in close proximity to the user 706. FIG. 8 illustrates the procedure for such comparison. The real-world object 714, which was recognized as a result of the comparison, can be outlined by a frame 716 or other visual means. The user 706 can view the message 106 displayed on the screen of the user device 717, interact therewith and/or interact with other users who created it, interacted therewith or will interact therewith in the future, using a user interface of software installed on the user device 717”; col.19. ; lines 11-21 “The data received (including metadata) by the user device from the server is compared by the user device with the processed data from the camera of the user device, and the user device thereby determines the visibility of a certain object 714. Mentioned object is then marked for user as being found in any fashion, and content is then added thereto in augmented reality (715, 717). Thus, the added content can be outlining with a frame, placing, in the augmented reality, visual content including text content, over or on the object, or other visual effects”; Fig. 6A-6B of Grant) In addition, the same motivation is used as the rejection for claim 1. Trim, Goodrich, Tiutiunnik and Grant understand to be silent on the remaining limitations of claim 5.
In the same field of endeavor, Freese teaches wherein the XR object corresponds to XR content that is object-locked to the real-world object ([0023] HMD 112 includes an image capture system 138 (e.g., cameras) for capturing image data of the surrounding physical environment and may also include one or more motion sensors (e.g., accelerometers) for tracking motion of the HMD 112. More specifically, as further described herein, image capture system 138 of HMD 112 capture image data representative of objects in the real world, physical environment that are within a field of view 130 of image capture system 138. Field of view 130 typically corresponds with the viewing perspective of HMD 112. HMD 112 may comprise a passthrough display configured to allow a user of HMD 112 to view the real-world, physical environment within the field of view of HMD 112 or may be configured to operate in a passthrough mode. In some examples, HMD 112 performs pose tracking of one or more real-world objects in the captured image data and renders artificial reality content (e.g., virtual objects) overlaid, entirely or in part, upon those one or more objects (e.g., locks virtual objects to real-world objects). AR calibration testing system 10 performs end-to-end AR system testing of the alignment of virtual objects to physical, real-world objects by HMD 112 without having to rely on subjective or inaccurate feedback from a human users”; [0025] As shown in FIG. 1, robotic apparatus 118 may include or have affixed thereon an image capture system 108 configured to capture images of a viewing area of HMD 112 from the perspective of a would-be user wearing HMD 112 (e.g., from the inside of the HMD 112 looking out into the real-world, physical environment surrounding the HMD 112). These images can include any artificial reality content displayed at HMD 112 and/or real-world objects that are visible through HMD 112 (e.g., objects that are in the field of view of HMD 112) or objects that are presented by HMD 112 in a passthrough mode (e.g., objects that are in field of view 130 of image capture system 138). In some examples, image capture system 108 may include two image captures devices corresponding to a user's eyes. In this way, AR calibration testing system 10 may test the accuracy with which the AR system presents, to a user's eyes, artificial content that is locked to real-world, physical objects. The term “locked” refers to artificial reality content that is presented at a location in the viewing area of the HMD that is relative to a physical object in the viewing area of the HMD. In some aspects, image capture system 138 and/or image capture system 108 may capture images in the visible light spectrum, the infrared spectrum, or other spectrum. Image processing described herein for identifying objects, object poses, and gestures, for example, may include processing infrared images, visible light spectrum images, and so forth. In some examples, image capture system 108 may comprise one or more RGB cameras.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of overlaying the specific tag onto the image based on text message description of Trim, Goodrich, Tiutiunnik and Grant with locking virtual objects to real world objects as seen in Freese because this modification would present virtual objects at a location in the viewing area that is relative to a physical object in the viewing area ([0025] of Freese)
Thus, the combination of Trim, Goodrich, Tiutiunnik, Grant and Freese teaches wherein the XR object corresponds to XR content that is object-locked to the real-world object.
Regarding claim 13, Trim, Goodrich, Tiutiunnik and Grant teach the device of claim 9, Remaining limitations of claim 13 is similar scope to claim 5 and therefore rejected under the same rationale.
Regarding claim 19, Trim, Goodrich, Tiutiunnik and Grant teach the non-transitory memory of claim 15, Remaining limitations of claim 19 is similar scope to claim 5 and therefore rejected under the same rationale
3. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Trim et al, U.S Patent Application Publication No 2020/0387537 (“Trim”) in view of Goodrich et al, U.S Patent Application Publication No.2022/0197027 (“Goodrich”) further in view of Tiutiunnik et al. U.S Patent No.11430211 (“Tiutiunnik”) further in view of Grant et al., U.S Patent Application Publication No. 20200401802 (“Grant”) further in view of Hisamitsu, U.S Patent Application Publication No.2006/0051063 (“Hisamitsu”)
Regarding claim 8, Trim, Goodrich, Tiutiunnik and Grant teach the method of claim 1, further comprising:
composing a subsequent electronic message including a different attachment associated with a different real-world object; and transmitting the subsequent electronic message to a recipient (see at least Fig.6a of Trim; [0020] of Goodrich “ The virtual conversation objects can include chat bubbles that include messages transmitted by respective participants of the conversation. The chat bubbles can also identify conversation participants using avatars, names or other identifiers unique to each participant. The virtual conversation objects can include interactive elements, such as a microphone button or option or a message generation option. The interactive elements allow the user to generate messages for transmission to other participants in the conversation”; see at least at col.10, lines 66-67-col.11, lines 1-61” of Tiutiunik “In a particular case, FIG. 2 shows the process of displaying, on the user device 109, an example of a tag that has been added to a real-world object 104 using the augmented reality means (technologies), with a social media content in form of a message associated with a real world object by Social Media user using the mentioned tag. In particular, FIG. 2 shows the visualization and functionality of such a message. The location of the real-world object 104 depicted on the user device 109 is outlined in FIG. 2 with a dashed frame 105. It should be noted that outlining the real-world object 104 can be performed either using the frame 105, or in any other fashion, for example, by contouring objects, shading, highlighting, or using other visual effects. A message area 203, comprising messages associated with the real-world object 104, in particular added (by Social Media users), contains message text 204, in particular a text or image of at least one message. In a particular case, the message area 203 may be located above the object or on the object itself, or in other way visually related to the object. The message area 203 may contain an avatar of a Social Media user 205, for example, mentioned avatar may be an image, in particular a photo, a picture, video, etc., for example, a user photo (portrait, etc.), and may also be a symbolic avatar or an emoji-avatar. It should be noted that the avatar can be a graphical and/or textual presentation of the user, user's alter ego, game character, internet character, application user (for example, the user of the application 102), etc., wherein the avatar can be a two-dimensional (static or animated) image (for example, an icon) and/or a three-dimensional model, and/or text…. A user-created message that was added (in particular attached) to a real-world object and can be displayed together with the information 208, interactive elements, avatars, message text, comment text, quantitative indicators 206, emojis 207, etc. or with a different set of elements that in combination fulfills a similar role”; col.13, lines 11-43 “In a particular case, the community is being formed after at least one subscription of at least one user for at least one different user, multiple users, social media content associated with (added to) a real-world object (or objects), hashtag, etc. For example, a user can join a social media content that is associated with (added to) real-world object, for example, a discussion, displayed on the user device. In a particular case, the interests that several users share that are related to real-world objects can be at least one gym, a gym network, gym attendance, a coffee shop, etc., or real-world objects that can be associated with any of those, or other objects. One of the users may visit the same establishment (gym, coffee shop, etc.), mentioned users can be relatives of each other (for example, children, wife, husband, father, mother, etc.), and so be engaged in separate social media interactions related to their family. It should be noted that at least one user can interact (using own device 109 mentioned herein) with at least one object of interest, including one that is different from any of objects of interest of a different user. For example, one user can interact, using own device (109) mentioned herein, with other objects of interest. Such objects of interest may include various real-world objects, such as ones mentioned herein: offices (405), educational institutions, such as schools (409), gyms (404), a donut, etc. For example, at least one user (or at least a portion of users), can be combined in at least one (the one described above or a different) community (group) depending on the interactions of the users with mentioned objects, in particular since such objects are interesting (or may (potentially) be interesting (become interesting) to at least one user using the invention described herein, in particular a Social Media user.”; col.13, lines 44-67-col.14, lines 1-34 “In a particular case, FIG. 5 shows an example of the real-time interaction between users of Social Media that happen to be nearby and are united by an event, point of interest, or phenomena that can be related to a visually distinctive physical object directly or indirectly. When one of the users (one of the Social Media users) sees the object of interest 501, the user, by using the user device 109 mentioned herein, can create a social media content in form of a message 506, and associate it with the mentioned object of interest. The message 506 can be displayed on at least one device of at least one Social Media user (101), in particular containing important (to the users) information. After adding mentioned message 506, at least one of the users (in particular at least one different user) who, in a particular case, is located nearby mentioned object of interest 501, can (immediately) join the conversation (thread) by pointing the camera of the user device on the same object of interest 501. If the message associated with the object of interest (and/or the real-world object) is out of direct visibility for the user, the user is able to join the conversation (thread) by pointing the user device camera at the direction of the object of interest. To join the thread, the user may also find a message with a tag in the list of new or nearest threads, or in another list, in particular sorted by a certain criterion, for example, by the criterion of creation or interaction recency, proximity, relevance, presence of certain user hashtags, etc. As a result of several interactions as one described here, a temporary community 503 can be created, which includes users who either do or do not know each other personally or from previous social media interactions, in particular the community can be created based on an interest in the same object of interest, for example, the object of interest 501 or a phenomenon. A social media user is able to join a conversation, thread, or a community, by interacting with a particular or several real-world object, or a social media content that is associated with a particular real-world object. Mentioned real-world objects can be used to join a particular a conversation, thread, or a community, are connected by relevance, visual similarities, and/or similar category, or are connected by actions of users ….The Social Media users can generate content, such as (text) messages, comments, emoji, images, videos or different media content, and can also share such content with different users, in particular with at least one community. It should be noted that the users can find at least one community and, in particular join such community, by searching for such a community by a keyword or hashtag. In a particular case, the hashtag can be a community-forming entity, i.e. (completely) determine the association of a tag to a community.” Where users can join the thread and generate content is considered composing a subsequent electronic message; [0106] of Grant “ With the match by the RU device 420′ of the particular object corresponding to an object stored in the TODB 482, in operation 545, the RU 450 may be presented with a text or graphic overlay of the message and possibly other information associated with the object (in the use case, the plant 610) by a message receiver 434′. This other information may include some or all of the additional information associated with the object. The RU 450 may switch roles with the TU 410 and add their own tagged information related to the object. In the use case, Zach may go ahead and water the plant 610, and then leave a message for Jordan using the procedure described above indicating, e.g., “I added two cups of water to the flowers. Is that enough?”. One or more of the original tags associated by the object may be updated or removed, as described above.) In addition, the same motivation is used as the rejection for claim 1. Trim, Goodrich, Tiutiunnik and Grant are understood to be silent on the remaining limitations of claim 8.
In the same field of endeavor, Hisamitsu teaches composing electronic message including a different attachment flag that associated with a different object; and transmitting the electronic message to a recipient ([0044] After completion of the record processing, an attachment flag is set to an on state. In addition, a sending screen (see FIG. 3(F)) including the file number of the image data and the text T2 is displayed on the LCD 38 by sending image display processing. [0045] When a sending operation is performed by the key input device 46, text sending processing that is noted to the text T2 is executed. That is, the CPU 28 performs packet processing by noting the text T2 to thereby send a mail message formed by a plurality of packets thus obtained to a desired destination through the radio circuit 14. [0046] After completion of the text sending processing, the CPU 28 notes the attachment flag. Since the attachment flag is an on state, the compressed image data created in response to the reply operation in order to attach the image file including the compressed image data of 480 horizontal pixels.times.640 vertical pixels to the text T2 sent to the desired destination is transferred from the flash memory 40 to the RAM 44, and then, attached image sending processing is executed. The CPU 28 performs packet processing on the compressed image data transferred to the RAM 44, and the mail message formed by a plurality of packets thus obtained is sent to the desired destination through the radio circuit 14. After completion of the sending processing, the attachment flag is set to an off state to end the reply task. [0047] A description is made on reply processing with respect to a browsing mail message attached with an image P3 having aspect ratio different from the initial value of the aspect ratio with reference to FIG. 5(A) and FIG. 5(B), and FIG. 6(A) and FIG. 6(B). With reference to FIG. 5(A), when a reply operation is performed while a browsing mail message including the image P3 (see FIG. 6(A)) showing the aspect ratio of 3:5 and having the "display manner" of 240 horizontal pixels.times.400 vertical pixels and a text T3 is displayed, a selection screen (see FIG. 3(B)) is displayed. [0048] When the "image creation" is selected, camera setting processing is executed, and an imaging task is activated. It is noted whether or not there is a related image relating to the browsing mail message. At this time, the image P3 is noted as a related image, and the camera setting that is forming a part of the EXIF information of the image P3 is written to the RAM 44. The "display manner" of the camera setting written to the RAM 44 shows 240 horizontal pixels.times.400 vertical pixels. Thus, image data of 240 horizontal pixels.times.400 vertical pixels is stored in the RAM 44 in response to the imaging task”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of overlaying the specific tag onto the image based on text message description of Trim, Goodrich, Tiutiunnik and Grant with including an attachment flag in sending message as seen in Hisamitsu because this modification would indicate whether attach the image file to the text and attached image sending processing is executed ([0046] of Hisamitsu)
Thus, the combination of Trim, Goodrich, Tiutiunnik, Grant and Hisamitsu teaches further comprising: composing a subsequent electronic message including a different attachment flag associated with a different real-world object; and transmitting the subsequent electronic message to a recipient.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH LE whose telephone number is (571)270-7842. The examiner can normally be reached Monday: 8AM-4:30PM EST, Tuesday: 8 AM-3:30PM EST, Wednesday: 8AM-2:30PM EST, Thursday and Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARAH LE/Primary Examiner, Art Unit 2614