Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
1. Applicant's election with traverse of claims 1-21, AND 25-26 in the reply filed on 08/27/2024 is acknowledged. The traversal is on the ground(s) that the inventions are not distinct as the inventions do not have a materially different design, mode of operation, function, or effect. This is found persuasive. The requirement is withdrawn. 1.
Claims 22-24 are canceled. Claims 1-21, and 25-26 are pending for examination.
2. Continuity: This application filed 01/06/2025 is a National Stage entry of PCT/EP2023/068303, International Filing Date: 07/04/2023 and PCT/EP2023/068303 Claims Priority from Application 17859833, filed 07/07/2022.
Claim Rejections - 35 USC § 101
3. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-21, 25-26 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more, when analyzed as per MPEP 2106.
Step 1 analysis:
Claims 1-20 are to a process comprising a series of steps, claims21 and 25-26 are to a device/server that is an apparatus, which are statutory (Step 1: Yes).
Step 2A Analysis:
Claim 1 recites:
A computer-implemented method for guiding a user interacting with an object, the method comprising:
(i) receiving informational cues of a scene including the object, the informational cues pertaining to sensor-retrieved data;
(ii) identifying the object based on the informational cues;
(iii) extracting contextual information based on the informational cues;
(iv) determining at least one interaction indicator configured to indicate an interaction performable on the identified object, the at least one interaction indicator being based at least in part on the contextual information; and
(v) guiding the user to perform the interaction on the object based on the at least one interaction indicator through control of a mixed reality MR device.
Step 2A Prong 1 analysis: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim.
Claims 1-21, 25-26 recite abstract idea.
The highlighted limitations of steps comprising, “ (ii) identifying the object based on the informational cues; (iii) extracting contextual information based on the informational cues; (iv) determining at least one interaction indicator configured to indicate an interaction performable on the identified object, the at least one interaction indicator being based at least in part on the contextual information”, fall within the mental process groupings of abstract ideas because they cover concepts performed in the human mind, including observation, evaluation, judgment, and opinion. See MPEP 2106.04(a)(2), subsection III. These limitations, as drafted, comprise a simple process, which, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of “a computer implemented method”. That is, other than reciting “by a computer implemented method” nothing in the claim elements precludes the steps from practically being performed in the mind. For example, but for the “by a computer implemented method” language, the claim encompasses a person looking at data comprising the informational cues of a scene , which can include images, video, or auditory data [See Specification para 0024], evaluating and forming simple judgements recognizing objects in the images, or video received, and determine contextual information, for example, if it is a bedroom scene, the contextual information could include on the bedroom furniture, and then can determine a required interaction to be taken as what item in the furniture to select to receive more information or indicate an intention to purchase. The mere nominal recitation of by a computer implemented method does not take the claim limitations out of the mental process grouping. Accordingly, the claim 1 recites a mental process. Since the other two independent clams 21 and 26 recite similar limitations as discussed for claim 1, they are analyzed on the same basis as recite mental process.
Thus, claims 1 with its dependent claims 2-20, claim 21 with its dependent claim 25 and 26 recite “Mental Processes’ grouping of abstract ideas.
Step 2A Prong 2 analysis: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).
Claims 1-21, 25-26: The judicial exception is not integrated into a practical application.
Claim 1 recites the additional limitations of using generic computer components comprising a generic computer device/server implementing the following steps:(i) receiving informational cues of a scene including the object, the informational cues pertaining to sensor-retrieved data; (ii) identifying the object based on the informational cues; (iii) extracting contextual information based on the informational cues; (iv) determining at least one interaction indicator configured to indicate an interaction performable on the identified object, the at least one interaction indicator being based at least in part on the contextual information; and (v) guiding the user to perform the interaction on the object based on the at least one interaction indicator through control of a mixed reality MR device.
The limitations :(i) receiving informational cues of a scene including the object,” are mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2106.05(g) (“whether the limitation is significant”). In addition, all uses of the recited judicial exceptions require such data gathering and output, and, as such, these limitations do not impose any meaningful limits on the claim. These limitations amount to necessary data gathering and outputting. See MPEP 2106.05.
Further, limitations (i), (ii), (iii) and (iv) are recited as being computer implemented. The computer is recited at a high level of generality. In limitation (i), the computer is used as a tool to perform the generic computer function of receiving data. See MPEP 2106.05(f). In limitations (ii), (iii) and (iv), the computer is used to perform an abstract idea, as discussed above in Step 2A, Prong One, such that it amounts to no more than mere instructions to apply the exception using a generic computer. See MPEP 2106.05(f).
The limitations in step “(v) guiding the user to perform the interaction on the object based on the at least one interaction through control of a mixed reality MR device is a generic computer function. The recitation of using a mixed reality MR device” provides nothing more than mere instructions to implement a generic computer function. The use of a MR device is recited in a nominal manner to perform a generic function of enabling a user to take an interaction based on received information about objects in an image. See MPEP 2106.05(f). MPEP 2106.05(f) provides the following considerations for determining whether a claim simply recites a generic computer function with the words “apply it” (or an equivalent), such as mere instructions to implement a generic function on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. The function of using a MR device of guiding the user to perform the interaction on the object does not provide any technical improvements in the functioning of a MR device but for a mere nominal recitation of using MR device for performing a generic computer function. Here, the recitation of “using a MR device” merely indicates a field of use or technological environment in which a generic computer function is performed. Although the additional element “using a MR device” limits the generic computer function of) guiding the user to perform the interaction on the object, this type of limitation merely confines the use of the generic computer function to a particular technological environment (MR device) and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h).
Even when considered individually and viewed in combination, these additional elements do not integrate the recited judicial exception into a practical application because they do not add any meaningful limits on practicing the abstract idea (Step 2A, Prong Two: NO), and the claim is directed to the judicial exception. (Step 2A: YES).
Since the limitations of the other two independent claims 21 and 26 are similar to the limitations of claim 1, they are analyzed on the same basis as directed to an abstract idea.
Dependent claims 2-3, 9, 11, 14, 15, 16, 19, 20 and 25 recite non-functional descriptive subject matter describing, what is contextual information, what interaction indicator i and the extracted information include, what are the material of the surfaces, who sells the object, what information cues include, whether the interaction indicator is a visual or auditory or haptic signal, and the MR device is either of a smart phone, or a computing tablet and therefore these additional elements do not integrate the recited judicial exception into a practical application because they do not add any meaningful limits on practicing the abstract idea. Dependent claims 4-7, 10, 12, 13 and 18 reciting limitations such as determining a surface of the object, a pose of the object, a distance of the object and lighting conditions of the scene, interaction indicators of the identified object and ranking those identified indicators, selecting a surface , identifying an installation error based on received information data, comparing surfaces fall , selecting an interaction indicator based upon customer reviews, as analyzed for base claim 1, fall within mental processes. Claim 8 recites using machine learning in a nominal manner to implement the abstract idea of ranking the interaction indicators, claims 10 and 13 also recite the limitations of sending data and claim 17 recites displaying 3d Graphic of an avatar which are non-significant extra-solution activity and do not integrate the recited judicial exception into a practical application because they do not add any meaningful limits on practicing the abstract idea.
Thus, the additional elements in all the pending claims 1-21, 25-26, even when considered individually and viewed in combination, do not integrate the recited judicial exception into a practical application because they do not add any meaningful limits on practicing the abstract idea (Step 2A, Prong Two: NO), and the claim is directed to the judicial exception. (Step 2A: YES).
Step 2B analysis: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05.
The claims 1-21, 25-26 do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Since claims are as per Step 2A are directed to an abstract idea, they have to be analyzed per Step 2B, if they recite an inventive step, i.e., the claims recite additional elements or a combination of elements that amount to “Significantly More” than the judicial exception in the claim.
As discussed above with respect to Step 2A Prong Two, the additional elements in the claims 1-21, 25-26 amount to no more than mere instructions to apply the exception using a generic-computer components, and generally linking the judicial exception to a particular technological environment or field of use. The same analysis applies here in 2B, i.e., mere instructions to apply the exception using a generic-computer components, and generally linking the judicial exception to a particular technological environment or field of use using a generic computer -components cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
as per MPEP 2106 , a conclusion that an additional element or elements is/are extra-solution activity, or are well-understood, conventional and routine activity in step 2A should be re-evaluated in step 2B. Here the receiving, acquiring, transmitting, and displaying steps were considered are extra-solution activity, or are well-understood, conventional and routine activity activities in step 2A and thus it is re-evaluated in step 2B to determine if it is more than what is well-understood, routine, conventional activity in the field.
The background of the example does not provide any indication that the computer components are anything other than a generic, off the shelf computer component and the Symantec, TLI, OIP Techs, Versata court decisions cited in MPEP 2106.05(d) (ii) indicate that mere receiving, acquiring, transmitting, and displaying steps using a generic computer is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is here).
Additional elements comprising receiving data, sending data, and displaying data were both found insignificant extra-solution activity in Step 2A, Prong Two, because they were determined to be insignificant limitations as necessary data gathering/transmitting/outputting /displaying/presenting data. However, a conclusion that an additional element is insignificant extra-solution activity in Step 2A, Prong Two should be re-evaluated in Step 2B. See MPEP 2106.05, subsection I.A. At Step 2B, the evaluation of the insignificant extra-solution activity consideration takes into account whether or not the extra-solution activity is well understood, routine, and conventional in the field. See MPEP 2106.05(g). The background of the Application does not provide any indication that the computer components are anything other than a generic, off the shelf computer component and the Symantec, TLI, OIP Techs, Versata court decisions cited in MPEP 2106.05(d) (ii) indicate that mere data gathering/ transmitting/ outputting/displaying/presenting/ data steps using a generic computer are well-understood, routine, conventional function when they are claimed in a merely generic manner (as it is here).
Accordingly, a conclusion that the gathering/transmitting/outputting /displaying/presenting data steps are well-understood, routine conventional activities are supported under Berkheimer Option 2. See MPEP 2106.05 (f) 2: Whether the claim invokes computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general-purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit).
Even when considered individually and in combination, the additional elements in claims 1-21, 25-26 represent mere instructions to implement an abstract idea or other exception on a computer and insignificant extra-solution activity, which do not provide an inventive concept. (Step 2B: NO).
Thus, pending claims 1-21, 25-26 are found patent ineligible.
Claim Rejections - 35 USC § 102
4. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 19-21, 25-26 are rejected under 35 U.S.C. 102 (a) (1) as being anticipated by EP 4002328 A1 cited in the IDS filed 01/06/2025, hereinafter EP’328.
Reference claim 1, EP;328 teaches a computer-implemented method for guiding a user
receiving informational cues of a scene including the object, the informational cues pertaining to sensor-retrieved data [see paragraphs 0023-0030: " The artificial assistance device 1000, ………comprises at least one set of sensors 1010 configured to acquire information related to the environment 20 and information related to the user 10. Preferably, the set of sensors 1010 is configured to acquire a first data set related to the environment 20 and a second data set related to the user 10…….. information related to the environment 20 comprises objects 30 arranged in the environment as well as information related to the environment itself as places of interest or otherwise. It should be noted that the objects 30 can be physical objects arranged in the environment 20 with which the user 10 can interact (directly by means of method indications: by acting on the objects 30 themselves and/or by receiving/sending information/commands). ……More preferably, the objects 30 also comprise elements of daily life, for example, present in a business, home, shipyard or workshop environment………It should be noted that, as will be clarified below, the information related to the environment 20 comprises information related to the same (e.g., places of interest) with which the user 10 can interact directly (e.g., by moving within and/or going from one place to another). ……. the information related to the user 10 comprises current and future information regarding the location of the user in the environment 20 (regardless of the positioning of the objects in the environment and/or augmented reality), the relative position of the user 10 with respect to one or more objects 30 arranged in the environment 20, one or more interactions of the user 10 with the environment 20 or with the objects 30, the actions of the user 10 in the environment 20……. the set of sensors 1010 is configured to acquire information such as audio, video, depth, thermal, geolocation and/or gestures of the environment 20 and of the user 10 as well as direct information from the IoT devices 1100 arranged in the environment 20 for the detection of data representative of possible health risks. Thus, the set of sensors 1010 is selected from video, audio, IMU, gazebo, gyroscope, GPS, thermal, depth, infrared, gesturing, hand pose, head pose, human pose, environmental sensors for detecting gases, fumes, fine dust, particulate matter, or a combination thereof. Such sensors also comprise communication modules configured to put the IoT devices 1100 in signal communication with the artificial assistance device 1000……the artificial assistance device 1000 comprises an augmented reality viewer 1020 configured to allow the user to see an augmented reality. It should be noted that the artificial assistance device 1000 allows both augmented reality and mixed reality vision by means of the augmented reality viewer 1020”];
identifying the object based on the informational cues [see para 0041, “cf. D1, paragraph 0041: "the step of generating a set of action signals 130 comprises the steps of recognizing 131 at the acquisition time instant one or more objects 30 present in the environment 20 as a function of the first data set"
"extracting contextual information based on the informational cues [see paras 0042 and 0043; [0042]: "the step of identifying 135 at the acquisition time instant a reference point PR present in the environment 20 as a function of the first data set", wherein "a reference point PR present in the environment" corresponds to the "contextual information" of claim 1. "];
determining at least one interaction indicator configured to indicate an interaction performable on the identified object, the at least one interaction indicator being based at least in part on the contextual information [See para 0043;"the step of generating a set of action signals 130 comprises the step of recognizing 139 at the acquisition time instant a type of action and/or interaction of the user 10 with the environment 20 among a plurality of types of reference actions and/or interactions as a function of the first data set and the second data set. Preferably, the type of action and interaction with the environment 20 comprises user interactions in the environment and/or with the objects 30 arranged in the environment itself. [...] Once the action and/or interaction at the acquisition time instant has been determined, the step of generating a set of action signals 130 includes determining 140 a future action and/or interaction of the user 10 with the environment 20 related to the time instant following the acquisition time instant. Thereby, the method not only determines the type of action and/or interaction at the acquisition time instant but is able to anticipate the action and/or interaction which the user must have with the environment 20 at a time instant following the acquisition time instant.". See para 0053: "Preferably, the step of determining an interaction instruction 160 includes determining the action and/or the interaction instruction based on the set of action and/or interaction signals combined with the first data set and the second data set. More preferably, the step of determining an action and/or interaction instruction 160 includes determining the action and/or interaction instruction as a function of the set of action and/or interaction signals combined with the first data set, the second data set, the set of environment signals, the set of action and/or interaction signals, or a combination thereof. Still more preferably, the step of determining an interaction instruction 160 includes determining the action and/or interaction instruction as a function of the set of action and/or interaction signals combined with the first data set, the second data set, the set of environment signals, the set of action and/or interaction signals, the reference instructions, or a combination thereof.". Also, see para 0055: "….. the step of determining an interaction instruction 160 comprises the step of drawing up 161 a list of actions to be taken by the user 10 as a function of the set of action signals. It should be noted that the list of actions which the user 10 must take comprises a list of actions with the environment 20, for example moving from one important zone to another, and with the objects 30 as well as with the IoT devices 1100. Preferably, the list of actions which the user 10 must take is a list of real actions to be actually performed with the environment 20 and/or with the objects 30."]; and
and guiding the user to perform the interaction on the object based on the at least one interaction indicator through control of a mixed reality (MR) device [See paras 0044--0045, 0050—0052, and 0054: " [0044]: "Preferably, the reference instructions comprise a list of actions and interactions with the environment 20, objects 30 as well as with the IoT devices which the user must perform for a given activity. More preferably, the reference instructions comprise threshold actions and interactions with the environment 20, objects 30 as well as with the IoT devices so as to generate an alarm signal when exceeded." ; [ 0045]: "Thereby, the method allows to anticipate an imminent hazard to the user 10, indicating the action and/or interaction with the objects 30 and/or the environment 20 to be carried out and/or to be avoided in order to prevent risks."; [para 0050: "Preferably, the method of artificial assistance 100 comprises the step of showing 170b the set of environment signals on the augmented reality viewer 1020."; Para 0051: "In accordance with a preferred embodiment, the method of artificial assistance comprises the step of generating 190 as a function of the first data set and the second data set by means of algorithms resident in the data processing unit a set of action and/or interaction signals representative of one or more user interactions 10 with the environment 20 related to the acquisition time instant. Preferably, the method of artificial assistance comprises the step of showing 170c the environment signal set on the augmented reality viewer 1020. More preferably, the step of showing the determined interaction signals on the augmented reality viewer 170c comprises the step of projecting 171c the interaction signals onto the augmented reality viewer 1020. In particular, the projecting step 171c includes signaling to the user the action and/or interaction in progress at the acquisition time instant."; [0052]: "In detail, an instruction for a real action and/or interaction with the environment 20 includes instructions to interact directly (physically and/or via signal) with the environment 20 (objects and/or environment itself) which the user can perform."; [0054]: Advantageously, the action and/or interaction instructions allow the user to give commands to the IoT devices, interact with the objects 30 and/or receive information related to safety and/or the activities carried out as well as to move within the environment between the important zones.".
Regarding claim 2, EP’328 teaches that the computer-implemented method of claim 1, wherein the contextual information includes one or more of a setting; a geographic location; a lifecycle period of the object; an environment type; an object defect; an object use or misuse; an object safety issue; and/or any combination thereof [See para 0029, “ set of sensors 1010 is configured to acquire information, such as audio, video, depth, thermal, geolocation and/or gestures of the environment 20 and of the user 10 “,.
Regarding claim 19, EP’ 328 teaches that the computer-implemented method of claim 1, wherein the informational cues include one of image data, mesh data, auditory data, tactile data, ambient data, motion data, olfactory data, or any combination thereof [See para 0029, “set of sensors 1010 is configured to acquire information, such as audio, video, depth, thermal, geolocation and/or gestures of the environment 20 and of the user 10 “.
Regarding claim 20, EP’ 328 teaches that the computer-implemented method of claim 1, wherein the at least one interaction indicator is a visual, auditory or haptic signal [See para 0029, “set of sensors 1010 is configured to acquire information, such as audio, video, depth, thermal, geolocation and/or gestures of the environment 20 and of the user 10 “, and also see para 0050].
Regarding claims 21, its limitations are similar to the limitations of claim 1 and is analyzed on the same basis as anticipated by EP”328.
Regarding claim 25, EP’328 teaches that the MR device can be one of smart phone, [see para 0002].
Regarding claims 26, its limitations are similar to the limitations of claim 1 and is analyzed on the same basis as anticipated by EP”328.
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5.1. Claim 3 and 15 is rejected under 35 U.S.C. 103 as being unpatentable over EP’ 328 in view of Nowak-Przygodzki et al. (US 20200342668), hereinafter Nowak.
Regarding claim 3, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system, but fails to teach that the computer-implemented method of claim 1, wherein the contextual information includes an object lifecycle period, the object lifecycle period including one of the object on display for sale; the object during installation; the object during use; the object at end of life, and/or any combination thereof. Nowak, in the same field of endeavor of determining of contextual data associated with the image received [see claim 1] and where the contextual data can include geolocation data and time data associated with the image feed [See Nowak para 0010], and further teaches [see para 0036 “[0036] One or more object identifiers can be generated by the image processing engine 114 and provided to the assistant application 118. The assistant application 118 can use the object identifiers to provide suggestions regarding the image conversation modes 120 that will be suggested to the user via the display device 108. For example, the assistant application 118 can store or access an index or table that correlates object identifiers to image conversation modes 120 (e.g., nutrition mode, price mode, fact mode, etc.) available to the assistant application 118. For instance, objects that are typically for sale, such as cars and food, can, for example, be correlated to a price mode. Also, for instance, food can additionally be correlated to a nutrition mode, whereas cars would not be correlated to the nutrition mode.]”. Therefore, in view of the teachings of Nowak in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of displaying items for sale as part of the images received from an environment/scene in EP’328, and secondly, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 15, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system, but fails to teach that the identified objects are on sale and sold by retailer. Nowak, as discussed above for claim 3 [see Nowak para [see para 0036 “[0036] One or more object identifiers can be generated by the image processing engine 114 and provided to the assistant application 118. The assistant application 118 can use the object identifiers to provide suggestions regarding the image conversation modes 120 that will be suggested to the user via the display device 108. For example, the assistant application 118 can store or access an index or table that correlates object identifiers to image conversation modes 120 (e.g., nutrition mode, price mode, fact mode, etc.) available to the assistant application 118. For instance, objects that are typically for sale, such as cars and food, can, for example, be correlated to a price mode. Also, for instance, food can additionally be correlated to a nutrition mode, whereas cars would not be correlated to the nutrition mode.] teaches displaying items for sale, wherein the items on sale could be cars or food and such items are sold by sellers and it would be obvious to an ordinary skilled in the art that the sellers of cars are car dealers [correspond to retailers] and sellers of food [correspond to food retailers].
5.2. Claims 4-10, 13-14, 16 are rejected under 35 U.S.C. 103 as being unpatentable over EP’ 328 in view of by Chojnacka et al. (US 20200342668), hereinafter Chojnacka.
Regarding claims 4-6, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system and guides the user to perform interaction on the object based at least one interaction indicator through control of mixed reality, but fails to teach the limitations, “4. The computer-implemented method of claim 1, wherein guiding the user comprises determining a surface of the object, and wherein the at least one interaction indicator is provided attached to the determined surface, claim 5 : determining a pose of the object and non-occluded surfaces of the object based on the informational cues being image and/or mesh data; and determining the surface based on the determined pose of the object and the determined non-occluded surfaces of the object, and claim 6: wherein the surface is further determined by: determining a distance to the object and lighting conditions of the scene, wherein determining the surface is based further on the distance to the object and the lighting conditions of the scene. Chojnacka teaches a computer-implemented method, in the same field of endeavor teaches detecting physical objects in a physical environment based on analysis of image frames captured by an image sensor of a computing device and recommending items, in particular, germane, pertinent, contextual items, for virtual placement, the method comprising and teaches the limitations of claim 4, “ wherein the AR device determines a surface of the object to present the recommendation in AR, and wherein the recommendation is presented attached to the determined surface [see para 0028]; “wherein the surface is determined by: determining a pose of the object and non-occluded surfaces of the object based on the stream of images of the scene; and determining the surface based on the determined pose of the object and the determined non-occluded surfaces of the object [0028], claim 6: wherein the surface is further determined by: determining a distance to the object and lighting conditions of the scene, wherein determining the surface is based further on the distance to the object and the lighting conditions of the scene [see para 0034]. Therefore, in view of the teachings of Chojnacka.in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of determining a surface of the identified object, and wherein the at least one interaction indicator is provided attached to the determined surface, determining a pose of the object and non-occluded surfaces of the object based on the informational cues being image and/or mesh data, and determining the surface based on the determined pose of the object and the determined non-occluded surfaces of the object, and claim 6: wherein the surface is further determined by: determining a distance to the object and lighting conditions of the scene, wherein determining the surface is based further on the distance to the object and the lighting conditions of the scene”, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claims 7-8, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system and guides the user to perform interaction on the object based at least one interaction indicator through control of mixed reality and determining a plurality of interaction indicators of the identified object, as analyzed for claim 1 above, but fails to teach the limitations, 7: ranking the plurality of interaction indicators, wherein the ranking is based at least in part on the contextual information and the determined at least one interaction indicator is the highest ranked interaction indicator among the plurality of interaction indicators and claim 8: wherein ranking the plurality of interaction indicators comprises processing the plurality of interaction indicators with a machine learning model to score the plurality of interaction indicators based at least in part on the contextual information. Chojnacka teaches a computer-implemented method in the same field of endeavor ranking the set of recommendations, wherein the ranking is based at least in part on the contextual information and the selected recommendation is the highest ranked recommendation [see para 0021] and wherein ranking the set of recommendations comprises: processing the set of recommendations with a machine learning model to score the set of recommendations based at least in part on the contextual information [see para 0021]. Therefore, in view of the teachings of Chojnacka.in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of ranking the plurality of interaction indicators, wherein the ranking is based at least in part on the contextual information and the determined at least one interaction indicator is the highest ranked interaction indicator among the plurality of interaction indicators and claim 8: wherein ranking the plurality of interaction indicators comprises processing the plurality of interaction indicators with a machine learning model to score the plurality of interaction indicators based at least in part on the contextual information, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 9, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, but fails to teach wherein the recommendation selected includes an object warning when the extracted contextual information includes an identified type of room in the scene that is not the type of room recommended for the object, wherein information pertaining to the type of room recommended for the object is obtained from a database. Chojnacka, in the same field of endeavor, as described above, teaches these missing limitations from EP’328 [see Chojnacka para 0040]. Therefore, in view of the teachings of Chojnacka.in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept that the recommendation selected includes an object warning when the extracted contextual information includes an identified type of room in the scene that is not the type of room recommended for the object, wherein information pertaining to the type of room recommended for the object is obtained from a database, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 10, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system and guides the user to perform interaction on the object based at least one interaction indicator through control of mixed reality, but fails to teach the limitations of claim 10: identifying at least two surfaces of the object; selecting a surface recommendation for each of the at least two surfaces; and sending the surface recommendations to the MR device, wherein each of the surface recommendations is presented in MR attached to a corresponding surface of the at least two surfaces. . Chojnacka teaches a computer-implemented method, in the same field of endeavor teaches detecting physical objects in a physical environment based on analysis of image frames captured by an image sensor of a computing device and recommending items, in particular, germane, pertinent, contextual items, for virtual placement, the method comprising and teaches the limitations of claim 10 [see Chojnacka para 0028].]. Therefore, in view of the teachings of Chojnacka.in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of identifying at least two surfaces of the object; selecting a surface recommendation for each of the at least two surfaces; and sending the surface recommendations to the MR device, wherein each of the surface recommendations is presented in MR attached to a corresponding surface of the at least two surfaces, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 13, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system and guides the user to perform interaction on the object based at least one interaction indicator through control of mixed reality, but fails to teach the limitations of claim 13: identifying surfaces of the object; comparing the surfaces on the object to models stored in a database to identify a defective surface; and sending a message to the MR device to present a recommendation related to the defective surface. Chojnacka teaches a computer-implemented method, in the same field of endeavor teaches detecting physical objects in a physical environment based on analysis of image frames captured by an image sensor of a computing device and recommending items, in particular, germane, pertinent, contextual items, for virtual placement, the method comprising and teaches the limitations of claim 11 [see Chojnacka para 0028]. Therefore, in view of the teachings of Chojnacka.in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of identifying at least two surfaces of the object; selecting a surface recommendation for each of the at least two surfaces; and sending the surface recommendations to the MR device, wherein each of the surface recommendations is presented in MR attached to a corresponding surface of the at least two surfaces, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 14, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system and guides the user to perform interaction on the object based at least one interaction indicator through control of mixed reality, but fails to teach the limitations of claim 14: wherein the at least one interaction indicator includes one or more of object materials; object dimensions; object care instructions; disposal information; sustainability information; assembly information; installation error information; consumable/replacement information; object intended use information; and/or any combination thereof. Chojnacka teaches a computer-implemented method, in the same field of endeavor teaches detecting physical objects in a physical environment based on analysis of image frames captured by an image sensor of a computing device and recommending items, in particular, germane, pertinent, contextual items, for virtual placement, the method comprising and teaches the limitations of claim 14 [see Chojnacka para 0028, “ [0028] As noted above, images of the physical environment 100A may be streamed in response to execution of an application running on the electronic device 110, and movement of the electronic device 110 and orientation of the camera of the electronic device 110 through the relevant portion of the physical environment 100A. As the corresponding image frames are processed through an algorithm executed by the application, the algorithm may detect a first series of surfaces 310A, for example, a series of adjacent horizontal and vertical surfaces. The algorithm may correlate the detected surface(s) 310A with shelves 310, based on the identification of books 315 positioned relative to the first series of surfaces 310A. Similarly, the system may detect a second surface 320A, for example, a horizontal surface, in the collected image frames, and correlate the detected second surface 320A with a desk 320, based on the identification of a keyboard 330 and a monitor 335 positioned on the second surface 320A, and a desk chair 340 positioned adjacent the surface 320A. In some implementations, physical items in the physical environment 100A such as, for example, the books 315, the keyboard 300, the monitor 335 and the chair 340, may be identified or recognized based on comparison and/or matching with images of previously identified items in a database accessible to the application. Based on the identification and/or recognition of these physical items, the system may identify surrounding items, context and/or functionality associated with the physical environment and the like. For example, the first series of horizontal and/or vertical surfaces may be identified as shelving based on the identification and/or recognition of the books 315 positioned thereon. Similarly, the second surface may be identified as a work surface, or desk (rather than, for example, a table) based on the identification and/or recognition of the keyboard 330 and the monitor 335 positioned thereon.]. Therefore, in view of the teachings of Chojnacka.in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of that the at least one interaction indicator includes one or more of object materials; object dimensions; object care instructions; disposal information; sustainability information; assembly information; installation error information; consumable/replacement information; object intended use information; and/or any combination thereof, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 16, EP’328 teaches all the limitations including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system and guides the user to perform interaction on the object based at least one interaction indicator through control of mixed reality, but fails to teach the limitations of claim 16: wherein the at least one interaction indicator is presented as 3D graphic on the MR device. Chojnacka teaches a computer-implemented method, in the same field of endeavor teaches detecting physical objects in a physical environment based on analysis of image frames captured by an image sensor of a computing device and recommending items, in particular, germane, pertinent, contextual items, for virtual placement, the method comprising and teaches the limitations of claim 14 [see Chojnacka para 0019, “ In general, this document describes example approaches for modeling spatial relations between objects in an ambient, or physical, or real world environment, and for providing automatic suggestion and/or placement of augmented, or mixed reality objects in the real world environment. In an augmented reality (AR) or a mixed reality (MR) system, or a virtual reality (VR) system, in accordance with implementations described herein, the system may analyze what is visible in the real-world environment, and provide for placement of three-dimensional (3D) virtual objects, or augmented/mixed reality objects, in a view of the real-world environment. For example, in some implementations, the system may analyze a stream of image information, to gain a semantic understanding of 3D pose information and location information related to real objects in the real-world environment, as well as object identification information related to the real objects, and other such information associated with real objects in the real-world environment. …….. With a semantic understanding of the real-world environment, suggestions for virtual object(s) to be placed in the real-world environment may be pertinent, or germane, or contextual, to the real-world environment, thus enhancing the user's experience.”]. Therefore, in view of the teachings of Chojnacka.in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of that the at least one interaction indicator is presented as 3D graphic on the MR device, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
5.3 Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over EP’ 328 in view of Chojnacka in view of Seroussi et al. (US 20250037395), hereinafter Seroussi.
The combined teachings of EP’ 328 in view of Chojnacka teaches and renders all the limitations of claim 10, as analyzed above, including wherein each of the at least two surfaces are made of a different material but fails to teach that the recommendation selected for each of the at least two surfaces includes care information for each of the different materials. Seroussi, in the same field of endeavor of receiving input images with 3 D features of scenes in an AR technology based computer implemented method , teaches [see para 0089] processing input/received images to identify 3D features which define the scene boundary (floor, wall, obstacles, objects etc.) and objects present therein to classify the scene type, style, theme (indoor, outdoor, room, bedroom, nursery, living room, outdoor veranda etc., the materials, color scheme and design style of the real world environment, e.g., modern, classic, industrial, gothic, etc.) and further teaches [see para 0122] that upon selection of the products for one or more of the designable areas, as seen in FIG. 5, the user is directed to an online shopping cart interface where they can proceed to purchasing one or more of the recommended products suitable to the user preferences and the listed items in the cart are optionally further provided with a “details” button which upon activation provides a pop-up screen with e.g., details about each item, instructions for care, placement, reviews, grading by other users, etc. Therefore, in view of the teachings of Seroussi, in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 in view Chojnacka to incorporate the concept that the recommendation provided [see claim 10] includes instructions for care for the selected item, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
5.4 Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over EP’ 328 in view of Bellaish et al., [US 20200151833 A1], hereinafter Bellaish.
Regarding claim 12, EP;328 teaches a computer-implemented method for guiding a user receiving informational cues of a scene [in the form of images] including the object, as analyzed for claim 1 above, but fails to teach further identifying an installation error based at least on the informational cues, the object, and the contextual information. Bellaish, in a similar field or endeavor of reporting construction errors detected from received images, teaches guiding a user receiving informational cues of a scene [in the form of images] including the object, and identifying an installation error based at least on the informational cues, the object, and the contextual information [See para 0024, “ at least part of an image captured from a construction site using an image sensor may be received “, and para 0186 “[0186] In some examples, the at least one electronic record (for example, the at least one electronic record updated by Step 1130, the at least one electronic record obtained by Step 920, etc.) may comprise information related to at least one construction error. For example, image data (such as image data captured from the construction site using at least one image sensor and obtained by Step 710) may be analyzed to identify at least one construction error related to the at least one object detected by Step 1120, and Step 1130 may record information related to the at least one construction error in the at least one electronic record. For example, Step 1120 may analyze the image data to identify an object installed incorrectly, and in response Step 1130 may record the incorrect installation of the object as a construction error in construction errors 640. ….., Step 1120 may identify a construction error based on the identified discrepancy, for example as described above, and Step 1130 may record the construction error identified by Step 930 in construction errors 640.”. Therefore, in view of the teachings of Bellaish, in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept of identifying an installation error based at least on the informational cues, the object, and the contextual information, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
5.5 Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over EP’ 328 in view of Chojnacka in view of Poltaretskyi et al. [US 20190380792 A1], hereinafter Poltaretskyi.
The combined teachings of EP’ 328 in view of Chojnacka teaches and renders all the limitations of claim 16, as analyzed above, but fails to teach that the 3D graphic presented on the MR device is an instructional avatar user virtually carrying out said interaction. Poltaretskyi, in the same field of using MR devices teaches that the MR device could be an avatar. Therefore, in view of the teachings of Poltaretskyi, in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 in view Chojnacka to incorporate the concept that the MR device could be an avatar., since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
.
5.6 Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over EP’ 328 in view of Perevodchikov et al., [US 11138631 B1], hereinafter Perevodchikov.
Regarding claim 12, EP;328 teaches a computer-implemented method including guiding a user to navigate through an environment/scene captured by a sensor including contextual information of the scene/environment captured, and where the objects of the scene/environment are identified by the system and guides the user to perform interaction on the object based at least one interaction indicator through control of mixed reality and determining a plurality of interaction indicators of the identified object, as analyzed for claim 1 above, but fails to teach wherein the at least one interaction indicator is selected based on at least one of customer reviews for the object and/or object return data. Perevodchikov, in the same field of endeavor of before selecting an interaction with a product reviewing customer reviews, teaches that at least one interaction indicator is selected based on at least one of customer reviews for the object [See col.7, line 46-col.8, line, “The user 100 may make a selection 132 of the option to view product reviews 126. A third user interface 128 may be presented with a number of product reviews for the wireless headphones. Based at least in part on the user's selection 132, the customer score determination engine 150 may determine that the last user action was to view product reviews, as shown at the third user interface 128. A timer may indicate that the user viewed the product reviews for 4 minutes, which may be an indication the user read the product reviews. ….”. Therefore, in view of the teachings of Perevodchikov in the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified EP’328 to incorporate the concept that the at least one interaction indicator is selected based on at least one of customer reviews for the object, since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Conclusion
6. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
(i) Scavezze et al. [US 20170004655 A1] cited in the IDS filed 01/06/2025 teaches a computer-implemented method for guiding a user interacting with an object comprising a method for interacting with a physical object in a mixed reality environment , receiving informational cues of a scene including the object, the informational cues pertaining to sensor-retrieved data [See para 0003], providing an HMD device operatively connected to a computing device with a display system for presenting a mixed reality environment and a camera for capturing images of physical objects, identifying the object based on the captured image [see para 0070], extracting contextual information based on the informational cues [see para 0071 “At 414 the method 400 includes determining an interaction context for the identified physical object based on one or more aspects of the mixed reality environment."], determining at least one interaction indicator configured to indicate an interaction performable on the identified object, the at least one interaction indicator being based at least in part on the contextual information [see para 0071], and guiding the user to perform the interaction on the object based on the at least one interaction indicator through control of a mixed reality (MR) device [see
(ii) Jindal et al. [US Patent# 11398089 B1; see Abstract] teaches techniques for identifying objects (such as products within a physical store) within a captured video scene and indicates which of object in the captured scene matches a desired object requested by a user, and then is displayed to the user in real-time via augmented reality. Object identification is carried out via a multimodal methodology.
. NPL reference:
(iii) N. J. Mitra, V. Kim, E. Yumer, M. Hueting, N. Carr and P. Reddy, "SeeThrough: Finding Objects in Heavily Occluded Indoor Scene Images," 2018 International Conference on 3D Vision (3DV), Verona, 2018, pp. 267-276, retrieved on 07/10/2026 describes scene graph generation algorithm with external knowledge and image reconstruction loss to refine object and phrase features for improving generalizability in scene graph generation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YOGESH C GARG whose telephone number is (571)272-6756. The examiner can normally be reached Max-Flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jeffrey A. Smith can be reached at 571-272-6763. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
YOGESH C. GARG
Primary Examiner
Art Unit 3688
/YOGESH C GARG/Primary Examiner, Art Unit 3688
Prosecution Insights