DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This Non-Final Office Action is in response to the RCE filed on11/17/2025. Claims 1-20 have been examined and are pending. Claims 1, 10, and 18 are amended.
Priority
Application 18/171,038 filed 02/17/2023 claims priority to provisional application 63/268,160 filed 02/17/2022.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/17/2025 has been entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. Claims 1-20 are directed to a system, method, or product which are/is one of the statutory categories of invention. (Step 1: YES).
Claims 1, 10 and 18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim recites a method and computing device for facilitating customer-agent interactions. For Claims 1, 10 and 18 the limitations of (Claim 1 being representative):
facilitating, […], an interaction between a user and an agent upon receiving a request for initiating an interaction from the user;
receiving[…], a […] workflow comprising a set of instructions from the agent, wherein the agent selects the […] workflow from a plurality of […] workflows based, at least in part, on interpreting a user objective for initiating the interaction;
receiving, […], a viewfinder frame […] associated with the user subsequent to initializing an […] session by the user in response to executing a first instruction from the set of instructions;
wherein viewfinder frames are neither stored nor forwarded to the agent to ensure that the user’s privacy is not compromised; and
iteratively performing, […], a plurality of operations until each instruction from the set of instructions is executed, the plurality of operations comprising:
[…] analyzing, […], the viewfinder frame to determine a subsequent instruction to be executed by the user from the set of instructions,
facilitating, […], a display of an […] image frame […], wherein the […] image frame is generated based, at least in part, on the subsequent instruction,
determining, […], an execution status of the subsequent instruction by monitoring the user while the user executes the subsequent instruction, the execution status indicating whether the subsequent instruction is one of successful and unsuccessful, and
transmitting[…], a notification indicating the execution status to the agent, as drafted, are processes that, under the broadest reasonable interpretation, covers certain methods of organizing human activity (i.e., commercial or legal interactions and/or managing personal behavior including following rules or instructions) but for recitation of generic computer components. The Examiner notes that “certain method[s] of organizing human activity” includes a person's interaction with a computer (see MPEP 2106.04(a)(2)(II)). That is, other than reciting a system implemented by an Augmented Reality (AR)- based workflow, electronic device, AR session, AR image, processor, memory, and non-transitory computer-readable storage medium, the claimed invention amounts to managing personal behavior or interaction between people and/or commercial or legal interactions. For example, but for the Augmented Reality (AR)- based workflow, electronic device, AR session, AR image, processor, memory, and non-transitory computer-readable storage medium, this claim encompasses a person to facilitate an interaction between a user and an agent upon receiving a request for initiating an interaction from the user, receiving a workflow comprising a set of instructions from the agent wherein the agent selects the workflow from a plurality of workflows based on interpreting a user objective for initiating the interaction, receive a frame associated with the user in response to executing an where the frames are neither stored nor forwarded, iteratively performing operations until each instruction is executed comprising: analyzing the frame to determine a subsequent instruction to be executed, displaying an image frame generated on the subsequent instruction, determining a status of the subsequent instruction by monitoring the user, the status indicating whether the subsequent instruction is one of successful and unsuccessful, and transmitting a notification indicating the execution status based on this data in the manner described in the identified abstract idea, supra. If a claim limitation, under its broadest reasonable interpretation, covers commercial or legal interactions and/or managing personal behavior or interactions between people but for the recitation of generic computer components, then it falls within the “certain methods of organizing human activity” grouping of abstract ideas. Accordingly, Claims 1, 10 and 18 recite an abstract idea. (Step 2A- Prong 1: YES. The claims recite an abstract idea).
This judicial exception is not integrated into a practical application. Claims 1, 10 and 18 recites the additional elements of a processing system (Claims 1), Augmented Reality (AR)- based workflow (Claims 1, 10 and 18), electronic device (Claims 1, 10 and 18), AR session (Claims 1, 10 and 18), AR image (Claims 1, 10 and 18), a processor (Claim 10, and 20), memory (Claims 10), a non-transitory computer-readable storage medium (Claim 18) that implements the identified abstract idea. These additional elements are not described by the applicant and are recited at a high-level of generality (i.e., one or more generic computers performing a generic computer functions) such that it amounts no more than mere instructions to apply the exception using a generic computer components. Accordingly, even in combination these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Claims 1, 10 and 18 are directed to an abstract idea. (Step 2A-Prong 2: NO: the additional claimed elements are not integrated into a practical application).
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of a processing system (Claims 1), Augmented Reality (AR)- based workflow (Claims 1, 10 and 18), electronic device (Claims 1, 10 and 18), AR session (Claims 1, 10 and 18), AR image (Claims 1, 10 and 18), a processor (Claim 10, and 20), memory (Claims 10), a non-transitory computer-readable storage medium (Claim 18), to perform the noted steps amounts to no more than mere instructions to apply the exception using a generic computer component (See Spec. Paragraphs [0027, 0031, 0034-0035]. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept (“significantly more’). Accordingly, even in combination, these additional elements do not provide significantly more. As such claims 1, 10 and 18 are not patent eligible. (Step 2B: NO. The claims do not provide significantly more).
Dependent Claims 2-9, 11-17 and 19-20 are similarly rejected because they either further define/narrow the abstract idea of independent claims 1, 10 and 18 as discussed above. Claim(s) 2 & 11 merely describe(s) an option to initialize the session. Claim(s) 6 & 15 merely describe(s) determining promotional content. Claims(s) 7 & 16 merely describe(s) overlaying instruction on the frame. Claim(s) 8, 17 & 20 merely describe(s) determining that a status is unsuccessful and displaying additional instructions. what the first set of information describing the potential candidate, second notification, and match with a request that the users recognize each other include. Therefore claims 2, 6-8, 11, 15-17, and 20 are considered patent ineligible for the reasons given above.
Dependent Claim(s) 3-5, 9, 12-14, and 19 recite limitations that further define the abstract idea noted in independent claims 1, 10, and 18. In addition, it recites the additional elements of a database, and virtual agent. The database, and virtual agent are recited at a high level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computing component. Even in combination, these additional elements do not integrate the abstract idea into a practical application and do not amount to significantly more than the abstract idea itself. Therefore, dependent claims 2-9, 11-17 and 19-20 are considered patent ineligible for the reasons given above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5, 8, 9, 10, 14, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Amir (US 20210174371 A1), in view of Avila (US 20180278750 A1), and in further view of Chachek (US 20200302510 A1) .
Regarding Claim 1,
Amir discloses, A computer-implemented method, the comprising: facilitating, by a processing system, an interaction between a user and an agent upon receiving a request for initiating an interaction from the user; " Support session 20 is initiated when a user 33 calls or otherwise contacts TSC 36 using a mobile device 31. Initiation (step A1) may be performed over a cellular and/or landline network, or other communication channels e.g., satellite communication, voice over IP, etc. When a call is received, TSC 36 sends a message (step A2) to the mobile device 31. The message may be SMS, email, WhatsApp, etc. and comprises a link (e.g. URL) for commencing a the support session. Upon opening the link (step A3), mobile device 31 accesses a remote server 36s over a data network 32, wherefrom video support session 21 setup instructions/code are sent to the mobile device 31 to establish the session (step A4). The remote server 36s may be implemented as part of the support center and/or in a cloud computing infrastructure accessible for both the users and the support center" (Amir Par. 0052).
receiving, by the processing system, an augmented reality (AR)-based workflow comprising a set of instructions from the agent, wherein the agent selects the AR-based workflow from a plurality of AR-based workflows based, at least in part, on interpreting a user objective for initiating the interaction; "Additionally, or alternatively, speech analysis tools may be used to analyze the user's speech to identify keywords within the speech and aid the computer vision tool as it processes image data 33i for relevant objects/elements within the image sensor field of view. For example, if the speech recognition tool identifies words such as internet/network and communication/connectivity in the user' speech, it may guide the computer vision tool to look for LAN sockets or cables, Wi-Fi antennas and/or LEDs indications. Optionally, the keywords may be typed by agent 36p. Upon identifying the relevant objects in the image data 33i, the TSC system using the computer vision tool can analyze the object's setup/configuration and automatically identify potential issues/defects therein. Display device 36d may be used to present to the identified object to the agent. Once the issue is identified, agent 36p may instruct the user on how to solve it. If the solution is relatively simple, (e.g., press the power switch), agent 36p may provide verbal instructions. If user 33 is unable to carry out the verbal instructions, or the instructions are relatively complex, agent 36p may generate an instructive augmented reality video stream using one or more markers 39 and trackers (step S8). The markers are superimposed onto the image data and displayed on the user's mobile device in real time to provide additional guidance. The agent may alternatively superimpose annotations as discussed and described above with respect to FIGS. 1A-1C. Optionally, TSC 36 may query database 36r for a best working solution (step S7), based on the object's determined issues/defects, and transmits the best working solution to user 33. The instructions may comprise textual, auditory and/or annotated/augmented content. Agent 36p may provide user 33 with some (including one) or all types of instructive content, and/or limit the content to include some or the entire set of instructions." (Amir Par. 0066-0068).
receiving, by the processing system, a viewfinder frame from an electronic device associated with the user subsequent to initializing an AR session by the user in response to executing a first instruction from the set of instructions; and "Upon establishing a video support session, the support center processes and analyzes the sounds and image data received from the remote end user. The support center provides tools for conveying instructions to the remote user. For example, where image data is comprised of one or more still images, the support center may add annotations, e.g., text, signs and/or symbols to the image data. Where image data is comprised of real-time video stream or video frames, the support center may superimpose a movable augmented indicator onto the image data. The annotated or superimposed image data is presented on the display of the user's mobile device. When the remote user successfully resolves the problem by following the annotated/superimposed instructions, the problem and solution is stored in a cloud or other database record system. By storing various problems and solutions, a database of working solutions is gradually established. The database may be used by the support center to more quickly and efficiently solve future problems. Alternatively, the database may form an artificial intelligence, whereby the artificial intelligence and not an agent of the support center solves the problem using the image data and relays the instructions to the user using annotations or a moveable augmented indicator as described above" (Amir Par. 0046).
iteratively performing, by the processing system, a plurality of operations until “FIG. 5B illustrates a decision tree used by support system 50 for determining the sequential instructions required to remedy the inoperability of an appliance. The input for the decision tree may be data derived from image data captured by image sensor 31c. For example, the image data may depict a current state of the inoperative product. The nodes in FIG. 5B represent operative states of the inoperative product and the lines represent the actions or the steps the user needs to complete to render the inoperative product operative.” (Amir Par. 0091). “If it is determined in step 50 that the best past solution obtained in steps 48-49 resolved the user's problem, a new database record 51 is constructed in step 54, and then stored in the database of the system for use in future trouble shooting sessions. The new database record may include one or more annotated images, a video showing how to fix the problem (with or without AR markers), text and/or audible instructions. If the best past solution is unsuccessful, other high-ranking solutions are obtained from the database, and presented in attempt to resolve the problem. Steps 48 to 50 may be repeated for each solution until a successful solution is found. Alternatively, or concurrently, speech analysis 12s, image recognition 12i, and steps 41 to 46 can be carried out to determine alternative problems/defects in the object 33e. (Amir Par. 0083).
electronically analyzing, by the processing system, the viewfinder frame to determine a subsequent instruction to be executed by the user from the set of instructions, “FIGS. 8A-8D illustrate another application of image processing system 60, wherein the image processing system 60 employs artificial intelligence during technical support. The image processing system comprises at least one processor configured to: receive image data captured by an image sensor of a mobile device, the image data including images of an inoperative appliance in an environment of a user; perform image recognition on the image data to determine an identity of the inoperative appliance and a likely cause of inoperability; retrieve a plurality of sequential instructions to be provided for enabling a user to complete a plurality of sequential actions in order to remedy the inoperability; cause the mobile device to sequentially display the plurality of sequential instructions; detect that the inoperative appliance is outside a field of view of the image sensor, based on the image data and during execution of the sequential actions; suspend display of additional sequential instructions while the inoperative appliance is outside of the field of view; detect when the inoperative appliance returns to the field of view after suspending display; and resume display of the additional sequential instructions after the inoperative appliance is detected to return to the field of view.” (Amir Par. 0122).
facilitating, by the processing system, a display of an AR image frame on the electronic device, wherein the AR image frame is generated based, at least in part, on the subsequent instruction, “As seen in FIG. 8A, once an appliance 83e and relevant functional elements 83f, 83h, and 83i are detected and identified in the live video stream, and a likely source of inoperability is assessed, a plurality of sequential instructions 85 for repairing/assembling the object are displayed on the mobile device. A likely source of error may be derived from database 36r as discussed above, or deduced from the operational state of the elements. For example, if the processor detects, using the data processing techniques described above, that the appliance is not connected to a power source, it may deduce that a lack of power is the cause of inoperability. Instructions 85 may be displayed one at a time, or multiple instructions may be displayed simultaneously. FIG. 8A illustrates yet another embodiment wherein at least two instructions and a moveable augmented indicator 84a are simultaneously displayed on a mobile device 81.” (Amir Par. 0123).
determining, by the processing system, an execution status of the subsequent instruction by monitoring the user while the user executes the subsequent instruction, the execution status indicating whether the subsequent instruction is one of successful and unsuccessful, and “Once retrieved, the processor annotates the image with one or more instructions for correcting the source of inoperability. FIG. 13B illustrates an annotated image wherein functional elements 103k, 103l have been identified as being in the wrong port. Annotation 134 therefore directs the user to switch the position of elements 103k and 103l to correct the configuration. Once corrected, the user may submit another image of the configuration to the processor, and the processor may notify the user that the step was completed successfully using positive feedback, e.g. check mark 135 (FIG. 13C).” (Amir Par. 0147). “The new database record may include one or more annotated images, a video showing how to fix the problem (with or without AR markers), text and/or audible instructions. If the best past solution is unsuccessful, other high-ranking solutions are obtained from the database, and presented in attempt to resolve the problem. Steps 48 to 50 may be repeated for each solution until a successful solution is found” (Amir Par. 0083).
Amir discloses facilitating an interaction between a user and an agent upon receiving a request for initiating an interaction from the user, receiving an augmented reality based workflow comprising a set of instructions from the agent, wherein the agent selects the AR-based workflow from a plurality of AR-based workflows based on interpreting a user objective for initiating the interaction, receiving a viewfinder frame from an electronic device associated with the user subsequent to initializing an AR session by the user in response to executing a first instruction from the set of instructions, iteratively performing a plurality of operations until each instruction from the set of instructions is executed, the plurality of operations comprising electronically analyzing the viewfinder frame to determine a subsequent instruction to be executed by the user from the set of instructions, facilitating a display of an AR image frame on the electronic device, wherein the AR image frame is generated based, at least in part, on the subsequent instruction, and determining an execution status of the subsequent instruction by monitoring the user while the user executes the subsequent instruction, the execution status indicating whether the subsequent instruction is one of successful and unsuccessful. Amir fails to disclose transmitting a notification indicating the execution status to the agent, and wherein viewfinder frames are neither stored nor forwarded to the agent to ensure that the user's privacy is not compromised. Avila, however, does disclose:
transmitting, by the processing system, a notification indicating the execution status to the agent. “The routing determination component 114 is configured to output a routing determination in accordance with a state of the help desk case. For instance, in an example where a help desk request is being created, an interaction between the matched support agent and the customer may be initiated through a modality of the help desk service. In an example, where a first support agent is involved in a communication with a customer, a second support agent (e.g. matched support agent) may be added/patched into the communication. In examples where a routing determination relates to generation of a follow-up inquiry for an unresolved help desk case, the routing determination component 114 may automatically provide a notification to a support agent to follow-up with a customer. For instance, an agent may be automatically assigned to follow-up with a customer for an unresolved request. In another instance where an agent is assigned to a help desk case, an agent may automatically receive a reminder to follow-up with a customer. In further examples, a follow-up inquiry may automatically be transmitted based to a customer and/or support agent. For instance, the routing determination model may identify that a follow-up is needed for a help desk case, evaluate presence information for the customer and/or support agent and transmit a communication. This processing may be useful in keeping resolution of the help desk case as a priority” (Avila Par. 0038).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined the image recognition of image data to determine an identity of the inoperative product and cause of the issue by retrieving instructions and providing visual guidance of Amir with transmitting a notification indicating the execution status to the Agent of Avila to identify that a follow-up is needed for a help desk case, and keep resolution of a case as a priority (Avila Par. 0038).
The combination of Amir and Avila fail to disclose wherein viewfinder frames are neither stored nor forwarded to the agent to ensure that the user's privacy is not compromised. Chachek, however, does disclose augmented reality navigating a location. Chachek discloses,
wherein viewfinder frames are neither stored nor forwarded to the agent to ensure that the user's privacy is not compromised. “In some embodiments, some or most of all of the computer vision analysis of captured image footage or video footage, may be performed locally within the end-user device of each such user of the crowd-sourced participants; thereby reducing the need and the cost to send and receive large amounts of image data or video data, or to store or process them at cloud-based servers or store-located servers; and/or increasing the level of privacy that is provided to users, as the system need not continuously share or upload a live stream of video or images, which involves a greater exposure to the privacy of the user himself as well as other in-store customers around him, but rather, the system only selectively uploads particular images or image-portions that depict a recognized product, rather than images of faces or people); and/or also increasing the speed of detecting products and localizing items within the store due to parallel computing or distributed computing efforts (e.g., ten end-user devices are performing locally and in parallel in near-real-time their computer vision analysis on their ten respective images or video streams; instead of the latency that would be created by each device having to upload its images or videos to a processing queue at a remote server)” (Chachek Par. 0012).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined the image recognition of image data to determine an identity of the inoperative product and cause of the issue by retrieving instructions and providing visual guidance of Amir and Avila with neither storing nor forwarding data to ensure that the user's privacy is not compromised of Chachek to maintain privacy and/or anonymity of users (Chachek Par. 0177).
Regarding Claim 10, and Claim 18
Amir discloses, An apparatus, comprising: at least one processor; and a memory having stored therein machine executable instructions, that when executed by the at least one processor, cause the apparatus, at least in part, to: (Amir Par. 0005)
A non-transitory computer-readable storage medium comprising computer-executable instructions that, when executed by at least a processor of an apparatus, cause the apparatus to perform a method comprising: (Amir Par. 0006)
facilitate an interaction between a user and an agent upon receiving a request for initiating an interaction from the user; “Support session 20 is initiated when a user 33 calls or otherwise contacts TSC 36 using a mobile device 31. Initiation (step A1) may be performed over a cellular and/or landline network, or other communication channels e.g., satellite communication, voice over IP, etc. When a call is received, TSC 36 sends a message (step A2) to the mobile device 31. The message may be SMS, email, WhatsApp, etc. and comprises a link (e.g. URL) for commencing a the support session. Upon opening the link (step A3), mobile device 31 accesses a remote server 36s over a data network 32, wherefrom video support session 21 setup instructions/code are sent to the mobile device 31 to establish the session (step A4). The remote server 36s may be implemented as part of the support center and/or in a cloud computing infrastructure accessible for both the users and the support center" (Amir Par. 0052).
receive an augmented reality (AR)-based workflow comprising a set of instructions from the agent, wherein the agent selects the AR-based workflow from a plurality of AR-based workflows based, at least in part, on interpreting a user objective for initiating the interaction; "Additionally, or alternatively, speech analysis tools may be used to analyze the user's speech to identify keywords within the speech and aid the computer vision tool as it processes image data 33i for relevant objects/elements within the image sensor field of view. For example, if the speech recognition tool identifies words such as internet/network and communication/connectivity in the user' speech, it may guide the computer vision tool to look for LAN sockets or cables, Wi-Fi antennas and/or LEDs indications. Optionally, the keywords may be typed by agent 36p. Upon identifying the relevant objects in the image data 33i, the TSC system using the computer vision tool can analyze the object's setup/configuration and automatically identify potential issues/defects therein. Display device 36d may be used to present to the identified object to the agent. Once the issue is identified, agent 36p may instruct the user on how to solve it. If the solution is relatively simple, (e.g., press the power switch), agent 36p may provide verbal instructions. If user 33 is unable to carry out the verbal instructions, or the instructions are relatively complex, agent 36p may generate an instructive augmented reality video stream using one or more markers 39 and trackers (step S8). The markers are superimposed onto the image data and displayed on the user's mobile device in real time to provide additional guidance. The agent may alternatively superimpose annotations as discussed and described above with respect to FIGS. 1A-1C. Optionally, TSC 36 may query database 36r for a best working solution (step S7), based on the object's determined issues/defects, and transmits the best working solution to user 33. The instructions may comprise textual, auditory and/or annotated/augmented content. Agent 36p may provide user 33 with some (including one) or all types of instructive content, and/or limit the content to include some or the entire set of instructions." (Amir Par. 0066-0068).
receive a viewfinder frame from an electronic device associated with the user subsequent to initializing an AR session by the user in response to executing a first instruction from the set of instructions; and "Upon establishing a video support session, the support center processes and analyzes the sounds and image data received from the remote end user. The support center provides tools for conveying instructions to the remote user. For example, where image data is comprised of one or more still images, the support center may add annotations, e.g., text, signs and/or symbols to the image data. Where image data is comprised of real-time video stream or video frames, the support center may superimpose a movable augmented indicator onto the image data. The annotated or superimposed image data is presented on the display of the user's mobile device. When the remote user successfully resolves the problem by following the annotated/superimposed instructions, the problem and solution is stored in a cloud or other database record system. By storing various problems and solutions, a database of working solutions is gradually established. The database may be used by the support center to more quickly and efficiently solve future problems. Alternatively, the database may form an artificial intelligence, whereby the artificial intelligence and not an agent of the support center solves the problem using the image data and relays the instructions to the user using annotations or a moveable augmented indicator as described above" (Amir Par. 0046).
iteratively perform a plurality of operations until “FIG. 5B illustrates a decision tree used by support system 50 for determining the sequential instructions required to remedy the inoperability of an appliance. The input for the decision tree may be data derived from image data captured by image sensor 31c. For example, the image data may depict a current state of the inoperative product. The nodes in FIG. 5B represent operative states of the inoperative product and the lines represent the actions or the steps the user needs to complete to render the inoperative product operative.” (Amir Par. 0091). “If it is determined in step 50 that the best past solution obtained in steps 48-49 resolved the user's problem, a new database record 51 is constructed in step 54, and then stored in the database of the system for use in future trouble shooting sessions. The new database record may include one or more annotated images, a video showing how to fix the problem (with or without AR markers), text and/or audible instructions. If the best past solution is unsuccessful, other high-ranking solutions are obtained from the database, and presented in attempt to resolve the problem. Steps 48 to 50 may be repeated for each solution until a successful solution is found. Alternatively, or concurrently, speech analysis 12s, image recognition 12i, and steps 41 to 46 can be carried out to determine alternative problems/defects in the object 33e. (Amir Par. 0083).
electronically analyze the viewfinder frame to determine a subsequent instruction to be executed by the user from the set of instructions, “FIGS. 8A-8D illustrate another application of image processing system 60, wherein the image processing system 60 employs artificial intelligence during technical support. The image processing system comprises at least one processor configured to: receive image data captured by an image sensor of a mobile device, the image data including images of an inoperative appliance in an environment of a user; perform image recognition on the image data to determine an identity of the inoperative appliance and a likely cause of inoperability; retrieve a plurality of sequential instructions to be provided for enabling a user to complete a plurality of sequential actions in order to remedy the inoperability; cause the mobile device to sequentially display the plurality of sequential instructions; detect that the inoperative appliance is outside a field of view of the image sensor, based on the image data and during execution of the sequential actions; suspend display of additional sequential instructions while the inoperative appliance is outside of the field of view; detect when the inoperative appliance returns to the field of view after suspending display; and resume display of the additional sequential instructions after the inoperative appliance is detected to return to the field of view.” (Amir Par. 0122).
facilitate a display of an AR image frame on the electronic device, wherein the AR image frame is generated based, at least in part, on the subsequent instruction, “As seen in FIG. 8A, once an appliance 83e and relevant functional elements 83f, 83h, and 83i are detected and identified in the live video stream, and a likely source of inoperability is assessed, a plurality of sequential instructions 85 for repairing/assembling the object are displayed on the mobile device. A likely source of error may be derived from database 36r as discussed above, or deduced from the operational state of the elements. For example, if the processor detects, using the data processing techniques described above, that the appliance is not connected to a power source, it may deduce that a lack of power is the cause of inoperability. Instructions 85 may be displayed one at a time, or multiple instructions may be displayed simultaneously. FIG. 8A illustrates yet another embodiment wherein at least two instructions and a moveable augmented indicator 84a are simultaneously displayed on a mobile device 81.” (Amir Par. 0123).
determine an execution status of the subsequent instruction by monitoring the user while the user executes the subsequent instruction, the execution status indicating whether the subsequent instruction is one of successful and unsuccessful, and “Once retrieved, the processor annotates the image with one or more instructions for correcting the source of inoperability. FIG. 13B illustrates an annotated image wherein functional elements 103k, 103l have been identified as being in the wrong port. Annotation 134 therefore directs the user to switch the position of elements 103k and 103l to correct the configuration. Once corrected, the user may submit another image of the configuration to the processor, and the processor may notify the user that the step was completed successfully using positive feedback, e.g. check mark 135 (FIG. 13C).” (Amir Par. 0147). “The new database record may include one or more annotated images, a video showing how to fix the problem (with or without AR markers), text and/or audible instructions. If the best past solution is unsuccessful, other high-ranking solutions are obtained from the database, and presented in attempt to resolve the problem. Steps 48 to 50 may be repeated for each solution until a successful solution is found” (Amir Par. 0083).
Amir discloses facilitating an interaction between a user and an agent upon receiving a request for initiating an interaction from the user, receiving an augmented reality based workflow comprising a set of instructions from the agent, wherein the agent selects the AR-based workflow from a plurality of AR-based workflows based on interpreting a user objective for initiating the interaction, receiving a viewfinder frame from an electronic device associated with the user subsequent to initializing an AR session by the user in response to executing a first instruction from the set of instructions, iteratively performing a plurality of operations until each instruction from the set of instructions is executed, the plurality of operations comprising electronically analyzing the viewfinder frame to determine a subsequent instruction to be executed by the user from the set of instructions, facilitating a display of an AR image frame on the electronic device, wherein the AR image frame is generated based, at least in part, on the subsequent instruction, and determining an execution status of the subsequent instruction by monitoring the user while the user executes the subsequent instruction, the execution status indicating whether the subsequent instruction is one of successful and unsuccessful. Amir fails to disclose transmitting a notification indicating the execution status to the agent and wherein viewfinder frames are neither stored nor forwarded to the agent to ensure that the user's privacy is not compromised. Avila, however, does disclose:
transmit a notification indicating the execution status to the agent. “The routing determination component 114 is configured to output a routing determination in accordance with a state of the help desk case. For instance, in an example where a help desk request is being created, an interaction between the matched support agent and the customer may be initiated through a modality of the help desk service. In an example, where a first support agent is involved in a communication with a customer, a second support agent (e.g. matched support agent) may be added/patched into the communication. In examples where a routing determination relates to generation of a follow-up inquiry for an unresolved help desk case, the routing determination component 114 may automatically provide a notification to a support agent to follow-up with a customer. For instance, an agent may be automatically assigned to follow-up with a customer for an unresolved request. In another instance where an agent is assigned to a help desk case, an agent may automatically receive a reminder to follow-up with a customer. In further examples, a follow-up inquiry may automatically be transmitted based to a customer and/or support agent. For instance, the routing determination model may identify that a follow-up is needed for a help desk case, evaluate presence information for the customer and/or support agent and transmit a communication. This processing may be useful in keeping resolution of the help desk case as a priority” (Avila Par. 0038).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined the image recognition of image data to determine an identity of the inoperative product and cause of the issue by retrieving instructions and providing visual guidance of Amir with transmitting a notification indicating the execution status to the Agent of Avila to identify that a follow-up is needed for a help desk case, and keep resolution of a case as a priority (Avila Par. 0038).
The combination of Amir and Avila fail to disclose wherein viewfinder frames are neither stored nor forwarded to the agent to ensure that the user's privacy is not compromised. Chachek, however, does disclose augmented reality navigating a location. Chachek discloses,
wherein viewfinder frames are neither stored nor forwarded to the agent to ensure that the user's privacy is not compromised. “In some embodiments, some or most of all of the computer vision analysis of captured image footage or video footage, may be performed locally within the end-user device of each such user of the crowd-sourced participants; thereby reducing the need and the cost to send and receive large amounts of image data or video data, or to store or process them at cloud-based servers or store-located servers; and/or increasing the level of privacy that is provided to users, as the system need not continuously share or upload a live stream of video or images, which involves a greater exposure to the privacy of the user himself as well as other in-store customers around him, but rather, the system only selectively uploads particular images or image-portions that depict a recognized product, rather than images of faces or people); and/or also increasing the speed of detecting products and localizing items within the store due to parallel computing or distributed computing efforts (e.g., ten end-user devices are performing locally and in parallel in near-real-time their computer vision analysis on their ten respective images or video streams; instead of the latency that would be created by each device having to upload its images or videos to a processing queue at a remote server)” (Chachek Par. 0012).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to have combined the image recognition of image data to determine an identity of the inoperative product and cause of the issue by retrieving instructions and providing visual guidance of Amir and Avila with neither storing nor forwarding data to ensure that the user's privacy is not compromised of Chachek to maintain privacy and/or anonymity of users (Chachek Par. 0177).
Regarding Claim 5, and Claim 14
The combination of Amir, Avila, and Chachek disclose the method of claim 1, as shown above. Amir, further discloses, The computer-implemented method of claim 1, wherein electronically analyzing the viewfinder frame further comprises: accessing, by the processing system, a plurality of images from a database associated with the processing system; “In some embodiments, TSC 36 is configured to record the video support sessions 21 in a database 36r. This way, the TSC 36 builds a continuously growing audio/visual database of user problems, and of their corresponding working solutions. The database may be used by computer vision tools to facilitate resolving of user's problems in future technical support sessions. Optionally, database 36r may be stored in a network computer/server of the TSC 36 (e.g., in the cloud)” (Amir Par. 0062). “Optionally, TSC 36 may query database 36r for a best working solution (step S7), based on the object's determined issues/defects, and transmits the best working solution to user 33” (Amir Par. 0068).
comparing, by the processing system, the viewfinder frame with the plurality of images; and “Control unit 12 is configured and operable to use image recognition module 12i to identify an object/functional element's set up/configuration and detect potential problems or defects therein. Database 36r can be used to store a plurality of erroneous setups/configurations (also referred to herein as reference data) to be compared by a comparison module 12u of control unit 12. Whenever the comparison module 12u determines that newly acquired image data 33i contains the same objects, issues or defects as the reference data, control unit 12 generates a diagnosis 12d identifying the erroneous setup/configuration identified in the image data 33i” (Amir Par. 0076).
upon determining a match between the viewfinder and an image from the plurality of images, determining, by the processing system, the subsequent instruction from the set of instructions based, at least in part, on the image. “Comparing new problems against a database of potentially related problems allows for precise problem identification. Once problems/defects are determined, database 36r is queried in step 48 for the best past solution as discussed above with respect to FIG. 2. After determining the best past solution, it is presented to the user via a display in the mobile device in step 49. The best past solution may include an annotated image, a video showing how to achieve the best problem solution (with or without AR markings), text and/or audible instructions, and is presented to the user via the display of the mobile device 31” (Amir Par. 0081).
Regarding Claim 8, and Claim 17
The combination of Amir, Avila, and Chachek disclose the method of claim 1, as shown above. Amir, further discloses, The computer-implemented method of claim 1, further comprising: upon determining that the execution status is unsuccessful generating, by the processing system, a set of intermediary instructions for rectifying the unsuccessful execution of the subsequent instruction; and “If the instructions are unsuccessful the user may be presented with an option to try again, or to receive human technical support. In the latter case, the processor may be configured to initiate a support session with a human operator when the processor determines that the completion of the plurality of sequential actions failed, or request that a technician be dispatched” (Amir Par. 0150).
facilitating, by the processing system, a display of another AR image frame on the electronic device, wherein the another AR image frame is generated based, at least in part, on the set of intermediary instructions. “The new database record may include one or more annotated images, a video showing how to fix the problem (with or without AR markers), text and/or audible instructions. If the best past solution is unsuccessful, other high-ranking solutions are obtained from the database, and presented in attempt to resolve the problem. Steps 48 to 50 may be repeated for each solution until a successful solution is found. Alternatively, or concurrently, speech analysis 12s, image recognition 12i, and steps 41 to 46 can be carried out to determine alternative problems/defects in the object 33e. If the problem is not resolved using a predefined number of database solutions, agent 36p may provide supplemental instructions or send a professional expert to the user 33 in step 52” (Amir Par. 0083).
Regarding Claim 9,
The combination of Amir, Avila, and Chachek disclose the method of claim 1, as shown above. Amir, further discloses, The computer-implemented method of claim 1, wherein the agent comprises at least one of a human and a virtual agent. “System 50 can be thus configured to concurrently conduct a plurality of support sessions 20, without any human intervention, using combined speech and image/video recognition techniques, to extract the proper and relevant keywords from auditory signals and/or image data obtained from user 33 that describe the experienced problem, and to determine the setup/configuration of the user's object” (Amir Par. 0100). “If the instructions are unsuccessful the user may be presented with an option to try again, or to receive human technical support. In the latter case, the processor may be configured to initiate a support session with a human operator when the processor determines that the completion of the plurality of sequential actions failed, or request that a technician be dispatched” (Amir Par. 0150).
Claim(s) 2, 3, 11, 12, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Amir (US 20210174371 A1), in view of Avila (US 20180278750 A1), in view of Chachek (US 20200302510 A1), and in further view of Waicberg (US 20210271882 A1) .
Regarding Claim 2, and Claim 11
The combination of Amir, Avila, and Chachek disclose the method of claim 1, as shown above. The combination of Amir, Avila, and Chachek fail to disclose a display of an option on the electronic device, the option enabling the user to initialize the AR session. Waicberg, however, does disclose The computer-implemented method of claim 1, further comprising: facilitating, by the processing system, a display of an option on the electronic device, the option enabling the user to initialize the AR session.