DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
This Office Action is in response to the Applicants’ Amendment/Remark filed on January 30, 2026. Applicant’s arguments, with respect to rejections of claims 1, 10 and 19 under 35 U.S.C 103(a) have been fully considered and are not persuasive because the combination of KOBEL and Schuster fairly discloses invention (see detail Office Action below). Claims 1, 4-5, 10, 13-14 and 19 have been amended. The KOBEL and Schuster’s references correctly do not teach claim 1 as now amended by Applicant, further consideration of the cited reference show alternate teachings and figures which do fully teach claim 1, as set forth in the new grounds of rejection. Inasmuch as the newly cited portions of the Sinha reference can be considered new grounds of rejection, such new grounds of rejection are necessitated by Applicant’s amendments to the claims. Claims 1-20 are still pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be supnegated by the manner in which the invention was made.
Claims 1-6, 10-15, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over KOBEL et al. (US 20230196688 A1) in view of Sinha et al. (US 20240403614 A1) and further in view of Schuster (US 20180336732 A1).
Regarding claim 1. KOBEL discloses an analytics computing device for analyzing building data to generate an image overlay for display with an image stream, the computing device comprising at least one processor in communication with at least one memory device, the at least one processor (KOBEL, see FIG. 15 and par. [0079]) configured to:
receive, from a user computing device, a request identifying (i) a building, (ii) an inquiry relating to the building, (iii) positional data relating to a position of the user computing device (KOBEL, see FIG. 2 and at least par. [0097] The application 1414 may be configured for inspecting, maintaining, troubleshooting and repairing the AR-compatible equipment 1406. In some embodiments, the AR troubleshooting module may overlay a 3D model or digital twin of the AR-compatible equipment 1406 over a live image of the real-world equipment 1406 on the screen of the user device. For example, the user 1402 may make a selection to enter an augmented reality portion of the application 1414. The user 1402 may then hold the user device so that a camera of the user device 1404 captures a live image of the equipment 1406 and the screen of the user device 1404 displays the live image. The application 1414 may then overlay the digital twin over the equipment 1406 and allow the user to view information about the equipment components as they look at the live image of the equipment 1406 on the screen of the user device 1404. The digital twin model may be partially or fully transparent or invisible.), and (iv) an image stream generated by a camera of the user computing device (KOBEL, see par. [0098] Referring now to FIG. 18, the user device 1404 is shown during operation of the application 1414, according to an example embodiment. Here, the user 1402 is holding the used device 1404 up to a real-world piece of equipment 1406. A camera of the user device 1404 may capture live images (e.g. video, a video stream, etc.) of the equipment 1406, and the live images 1902 may be displayed on the display screen 1502.);
in response to receiving the request, retrieve building data relating to the building from the at least one memory device, the building data including a floorplan of the building and a location of a plurality of components included within the building (KOBEL, see at least par. the user 1402 may more easily identify the corresponding components on the real-world equipment 1406. In some embodiments, the digital twin 1512 may update based on the live image 1902 of the real-world equipment. For example, a particular hydraulic hose may have been installed such that it hangs in a slightly different position than the corresponding hose was modeled in the digital twin CAD model. The application 1414 may detect the position of the real-world hose based on its being in a similar position to the digital twin hose and may update the digital twin such that the digital twin hose is more accurately overlaid on the real-world hose. Thus, when the digital twin hose is displayed and emphasized, it will more closely overlap with the live image of the real-world hose. In some embodiments, the application 1414 may be configured to detect that a component of the real-world equipment 1406 is significantly out of position, indicating that it may have been installed incorrectly or has moved out of position. In response to detecting that a component is out of position, the application 1414 may display an alert on the GUI 1506 indicating that the component needs attention, and may provide instructions for reinstalling the component. These features are discussed below in reference to FIG. 21.);
identify, based upon the inquiry, one or more target components of the plurality of components of the building, the target components relating to the inquiry (KOBEL, see par. [0100] As an example of the foregoing, a user 1402 may use the application to select the digital twin 1512 of the equipment 1406 the user 1402 wishes to inspect. The application 1414 may load the digital twin from the database 1410 onto the user device 1404. The user may select an option to enter augmented reality mode using the selected digital twin 1512. The user 1402 may point the camera of the user device 1404 toward the AR-compatible equipment 1406, causing the application 1414 to display a live image 1902 of the equipment 1406. The application 1414 may then overlay an invisible digital twin over the live image 1902 of the equipment 1406. The user 1402 may then select components by touching the live image 1902 on a touchscreen 1504 of the user device 1404, causing the application to display information about the selected component (e.g. sensor readings, part numbers, component service age, etc.). The user 1402 may determine whether any of the selected components needs to be repaired or replaced. The user may then move on to each other component to perform a complete inspection of the equipment 1406.);
generate, based upon the positional data and the building data, an image overlay including one or more indicators corresponding to locations of the one or more target components (KOBEL, see a least par. [0069] Augmented reality (AR) may be used to place digital objects into a live video feed of the real world and display the combined video feed on a display screen of a user device (e.g., a tablet computer, smartphone, laptop, smart TV, head-mounted display, etc.). In some embodiments of the present application, an AR application may overlay (e.g., superimpose, align, etc.) a digital twin of a piece of equipment onto a live video feed of the equipment. This can allow sensor measurements to appear overlaid onto the live video feed of the real-world equipment and can allow a user to select a component of the equipment on the live video feed and see more information about the component. Various components of the digital twin can be shown and emphasized on the live video feed so a user may be able to better identify and inspect the components. The AR application may allow a user to select a function of the equipment and the AR application may display and emphasize components of the digital twin associated with the selected function. Instructions for inspecting and repairing the equipment can be overlaid on the live video feed with visual indicators that can make the inspection and repair easier for the user. The AR application may be able to automatically detect the identity of the equipment and load the digital twin by recognizing the shape of the equipment or a decal affixed to the equipment.).
cause the user computing device to display the image stream with the generated image overlay (KOBEL, see pars. [0098] Referring now to FIG. 18, the user device 1404 is shown during operation of the application 1414, according to an example embodiment. Here, the user 1402 is holding the used device 1404 up to a real-world piece of equipment 1406. A camera of the user device 1404 may capture live images (e.g. video, a video stream, etc.) of the equipment 1406, and the live images 1902 may be displayed on the display screen 1502... [0100] As an example of the foregoing, a user 1402 may use the application to select the digital twin 1512 of the equipment 1406 the user 1402 wishes to inspect. The application 1414 may load the digital twin from the database 1410 onto the user device 1404. The user may select an option to enter augmented reality mode using the selected digital twin 1512. The user 1402 may point the camera of the user device 1404 toward the AR-compatible equipment 1406, causing the application 1414 to display a live image 1902 of the equipment 1406. The application 1414 may then overlay an invisible digital twin over the live image 1902 of the equipment 1406. The user 1402 may then select components by touching the live image 1902 on a touchscreen 1504 of the user device 1404, causing the application to display information about the selected component (e.g. sensor readings, part numbers, component service age, etc.). The user 1402 may determine whether any of the selected components needs to be repaired or replaced. The user may then move on to each other component to perform a complete inspection of the equipment 1406.).
KOBEL does not disclose in response to receiving the request, retrieve building data relating to the building from the at least one memory device, the building data including a floorplan of the building and a location of a plurality of components included within the building; and generate, based upon the positional data and the building data, an image overlay including one or more indicators corresponding to locations of the one or more target components. However,
Sinha discloses:
in response to receiving the request (Sinha, see at least par. The system can perform root cause prediction by being trained using data that includes indications of root causes of faults or errors, where the indications are labels for or otherwise associated with (unstructured or structure) data such as service requests, service reports, service calls, etc. The system can receive, from a service technician in the field evaluating the issue with the equipment, feedback regarding the accuracy of the root cause predictions, as well as feedback regarding how the service technician evaluated information about the equipment (e.g., what data did they evaluate; what did they inspect; did the root cause prediction or instructions for finding the root cause accurately match the type of equipment, etc.), which can be used to update the root cause prediction model.), retrieve building data relating to the building from the at least one memory device (Sinha, see at least par. [0089] In some embodiments, the digital occupant assistant application 120 may assist one or more users to improve their comfort in the building. For example, the digital occupant assistant application 120 may receive a query from a user asking, “why is hot in this room?” and the digital occupant assistant application 120 may retrieve a control schedule for the room. In this example, the digital occupant assistant application 120 may detect differences between the control schedule and a setpoint of the room. For example, the digital occupant assistant application 120 may detect that a current temperature of the room exceeds a temperature setpoint for the room. In some embodiments, the digital occupant assistant application 120 may generate a maintenance request to have a technician look in a potential cause.), the building data including a floorplan of the building and a location of a plurality of components included within the building (Sinha, see at least par. [0061] In some implementations, the training data includes data relating to building management systems. For example, the training data can include examples of HVAC-R data, such as operating manuals, technical data sheets, configuration settings, operating setpoints, diagnostic guides, troubleshooting guides, user reports, technician reports. In some implementations, the training data used to configure the first model 104 includes at least some publicly accessible data, such as data retrievable via the Internet. [0062] In some embodiments, the training data can include data relating to components of buildings. For example, the training data can include examples of floor layouts, such as blueprints, building spec, floorplans, BIMs, and how given components of a building can be orientated and/or positioned throughout the building.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of KOBEL, with in response to receiving the request, retrieve building data relating to the building from the at least one memory device, the building data including a floorplan of the building and a location of a plurality of components included within the building, as provided by Sinha. The modification provides an improved system and method for generating augmented reality interfaces providing locations of components within a building, thereby to improve the accuracy and quality of generated outputs and update training of the machine learning models accordingly. The system can enable real-time messaging and/or conversational interfaces for users to provide field data regarding equipment to the system (including presenting targeted queries to users that are expected to elicit relevant responses for efficiently receiving useful response information from users) and guide users, such as service technicians, through relevant service, diagnostic, troubleshooting, and/or repair processes. (Sinha, see par. [0044]).
KOBE in view of Sinha does not discloses generate, based upon the positional data and the building data, an image overlay including one or more indicators corresponding to locations of the one or more target components. However,
Schuster discloses:
generate, based upon the positional data and the building data, an image overlay including one or more indicators corresponding to locations of the one or more target components (see at least par. [0030] For example, another project may include installation of plumbing in a sink cabinet. The tasks identified for the project may include, for example, building or transferring plumbing from an old unit, cutting out the back of a cabinet, connecting the plumbing to a fixture, and other tasks. When performed manually, the project is labor intensive as many measurements must be performed, for example, on the back of the cabinet and inside the cabinet to confirm the location for the plumbing. In various embodiments described herein, a remote computing environment confirms the location of plumbing by using dimensions of a known item or location, such as a wall or a cabinet, and extrapolating those dimensions to programmatically determine dimensions of the cabinet. An augmented reality session may include overlaying graphics where the cabinet will be positioned, accounting for counter top, over hang, etc. The location of the drain line and water lines may be shown, for example to assist the operator in determining the correct size and location of a hole to be made in the back of the cabinet.); and
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of KOBEL, with generate, based upon the positional data and the building data, an image overlay including one or more indicators corresponding to locations of the one or more target components, as provided by Schuster. The modification provides an improved system and method for generating augmented reality interfaces providing locations of components within a building, thereby to reduc[ing] required skill or expertise, minimizing labor time, optimizing use of materials, and other benefits in construction, manufacturing, and related industries. (Schuster, see par. [0025]).
Regarding claim 2. KOBEL in view of Sinha and further in view of Schuster discloses the analytics computing device of claim 1 (as rejected above), KOBEL in view of Sinha and further in view of Schuster further discloses wherein the at least one processor is further is in communication with a plurality of sensors disposed in the building, and wherein the at least one processor is further configured to receive sensor data generated by the plurality of sensors (KOBEL, see at least par. [0075] As an example of the foregoing, the application 1414 may use the sensor information to adjust the digital twin such that the digital twin is in the same physical configuration as the AR-compatible equipment 1406. For example, if position sensors detect that the boom of a boom lift is extended 20 feet at a 25 degree angle, the application 1414 may update the digital twin such that the boom of the digital twin is also extended 20 feet at a 25 degree angle. Other sensor information (e.g. pressure, voltage, etc.) may be displayed on the screen 1502 in the location of the sensor. In other embodiments, the user 1402 may select a component, causing the application to display any sensor information relating to the selected component. For example, the user 1402 may select a hydraulic hose on the digital twin by touching the hose on the touchscreen 1504 of the user device 1404. In response to the selection, the application may update the digital twin to display the pressure sensor information for the selected hose. In addition to the functions discussed in this section, the AR application 1414 may also be configured to perform the functions of the AR application 114 discussed above. Similarly, the AR application 114 may be configured to perform the functions of the AR application 1414 discussed herein.).
Regarding claim 3. KOBEL in view of Sinha and further in view of Schuster discloses the analytics computing device of claim 2 (as rejected above), and KOBEL in view of Sinha and further in view of Schuster further discloses wherein the at least one processor is further configured to:
identify at least one warning condition in the building based upon the sensor data; and cause the user computing device to display a notification including the identified at least one warning condition (KOBEL, see at least par. [0103] Referring now to FIG. 21, the user device 1404 is shown during operation of the application 1414, according to an example embodiment. The defect detection module 1568 may detect a defect in the equipment 1406 and may display an alert on the GUI 1506 indicating that the defect has been detected. Defects may include that a component of the equipment 1406 has been installed incorrectly, is missing, or is otherwise out of place. The application 1414 may detect that a component is out of place if the component does not align with the corresponding component of the digital twin 1512 within a specified margin of error. In some embodiments, if a component of the equipment 1406 does not perfectly align with the corresponding component of the digital twin 1512, but is within a predefined margin of error, the application may adjust the digital twin 1512 such that the components are aligned. However, if the position of the real-world component falls outside the margin of error, the application 1414 may detect a defect and may display an alert that the component is out of place or missing. The application 1414 may further display instructions for fixing the defect and/or replacing the component.).
Regarding claim 4. KOBEL in view of Schuster discloses the analytics computing device of claim 3 (as rejected above), KOBEL in view of Schuster further discloses wherein the at least one processor is further configured to identify the target components of the plurality of components of the building further based upon the identified at least one warning condition (KOBEL, see par. [0104] In the example shown in FIG. 21, the application has detected that hydraulic hose 1918 is missing. The application displays an alert 2102 indicating that the hydraulic hose 1918 is missing. The application also displays instructions 2104, 2108, 2112 for installing a replacement hydraulic hose 1918. The first instruction 2104 instructs the user 1402 to connect the first end of hydraulic hose 1918 to the hydraulic cylinder 2120. The first instruction 2104 may have an associated arrow 2105 directing the user to the location that the hose 1918 is to be connected) and the building data retrieved from the at least one memory device (Sinha, see at least par. [0115] As depicted in FIG. 2, the system 200 can include a prompt management system 228. The prompt management system 228 can include one or more rules, heuristics, logic, policies, algorithms, functions, machine learning models, neural networks, scripts, or various combinations thereof to perform operations including processing data from data repository 204 into training data for configuring various machine learning models. For example, the prompt management system 228 can retrieve and/or receive data from the data repository 204, and determine training data elements that include examples of input and outputs for generation by machine learning models, such as a training data element that includes a prompt and a completion corresponding to the prompt, based on the data from the data repository 204.);
perform a lookup using the at least one warning condition and the target components associated with the at least one warning condition (Sinha, see at least par. [0092] In some embodiments, the digital occupant assistant application 120 may provide one or more prompts to a user. For example, the digital occupant assistant application 120 may provide prompts to proactively assist the user. As another example, the digital occupant assistant application 120 may provide prompts to alert the user of a change in a building. The alerts may include displaying banners or flags on a display device. In some embodiments, the digital occupant assistant application 120 may provide different prompts to one or more users based on a given piece of information. For example, a sink in a given restroom may be leaking water. In this example, the digital occupant assistant application 120 may provide a first prompt to inform a visitor of the building to utilize a different restroom. The digital occupant assistant application 120 may provide a second prompt to direct a technician of the building to address the leak.); and
cause the user computing device to display the image stream with the generated image overlay that includes (i) one or more indicators pointing to the target components associated with the at least one warning condition, and (ii) a notification describing the at least one warning condition (KOBEL, see at least par. [0069] [0069] Augmented reality (AR) may be used to place digital objects into a live video feed of the real world and display the combined video feed on a display screen of a user device (e.g., a tablet computer, smartphone, laptop, smart TV, head-mounted display, etc.). In some embodiments of the present application, an AR application may overlay (e.g., superimpose, align, etc.) a digital twin of a piece of equipment onto a live video feed of the equipment. This can allow sensor measurements to appear overlaid onto the live video feed of the real-world equipment and can allow a user to select a component of the equipment on the live video feed and see more information about the component. Various components of the digital twin can be shown and emphasized on the live video feed so a user may be able to better identify and inspect the components. The AR application may allow a user to select a function of the equipment and the AR application may display and emphasize components of the digital twin associated with the selected function. Instructions for inspecting and repairing the equipment can be overlaid on the live video feed with visual indicators that can make the inspection and repair easier for the user. The AR application may be able to automatically detect the identity of the equipment and load the digital twin by recognizing the shape of the equipment or a decal affixed to the equipment.).
Regarding claim 5. KOBEL in view of Sinha and further in view of Schuster discloses the analytics computing device of claim 1 (as rejected above), and KOBEL in view of Sinha and further in view of Schuster further discloses wherein the at least one processor is further configured to identify the building based upon the positional data (Schuster, see at least par. [0030] For example, another project may include installation of plumbing in a sink cabinet. The tasks identified for the project may include, for example, building or transferring plumbing from an old unit, cutting out the back of a cabinet, connecting the plumbing to a fixture, and other tasks. When performed manually, the project is labor intensive as many measurements must be performed, for example, on the back of the cabinet and inside the cabinet to confirm the location for the plumbing. In various embodiments described herein, a remote computing environment confirms the location of plumbing by using dimensions of a known item or location, such as a wall or a cabinet, and extrapolating those dimensions to programmatically determine dimensions of the cabinet. An augmented reality session may include overlaying graphics where the cabinet will be positioned, accounting for counter top, over hang, etc. The location of the drain line and water lines may be shown, for example to assist the operator in determining the correct size and location of a hole to be made in the back of the cabinet.),
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of KOBEL, with wherein the at least one processor is further configured to identify the building based upon the positional data, as provided by Schuster. The modification provides an improved system and method for generating augmented reality interfaces providing locations of components within a building, thereby to reduc[ing] required skill or expertise, minimizing labor time, optimizing use of materials, and other benefits in construction, manufacturing, and related industries. (Schuster, see par. [0025]).
and retrieve building data describing the identified building based upon the identified building (Sinha, see at least par. [0174] The generative LLM can also generate the data based on context relating to the at least one of the occupant, the building, a space of the building, an event relating to the building, one or more other occupants, or building equipment or other assets of the building. For example, the generative LLM retrieve, receive, detect, identify, and/or determine an identification of the occupant (e.g., a name of the occupant, an employee number, an employee ID, etc.) and the generative LLM can generate the data based on the identification of the occupant.)
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of KOBEL, with and retrieve building data describing the identified building based upon the identified building, as provided by Sinha. The modification provides an improved system and method for generating augmented reality interfaces providing locations of components within a building, thereby to improve the accuracy and quality of generated outputs and update training of the machine learning models accordingly. The system can enable real-time messaging and/or conversational interfaces for users to provide field data regarding equipment to the system (including presenting targeted queries to users that are expected to elicit relevant responses for efficiently receiving useful response information from users) and guide users, such as service technicians, through relevant service, diagnostic, troubleshooting, and/or repair processes. (Sinha, see par. [0044]).
Regarding claim 6. KOBEL in view of Schuster discloses the analytics computing device of claim 1 (as rejected above), and KOBEL in view of Schuster further discloses wherein the building data further includes one or more of instructions, images, and/or videos, and wherein the at least one processor is further configured to: identify at least one or more of the instructions, images, and/or videos based upon the request; and cause the user computing device to display the identified one or more of the instructions, images, and/or videos. (Schuster, see at least par. [0045] The 3D navigation window 210 may also include a maintenance manual button 223. Selection of the maintenance manual button 223 may cause the application 114 to update the GUI 206 to replace all or a portion of the GUI with a text-based maintenance manual (e.g., a portable document format (PDF) file, etc.) for the building equipment corresponding to the 3D model. In some embodiments, a new GUI (e.g. a new window, a new tab, etc.) may be generated, while the GUI 206 is preserved. The user may be able to toggle between the new GUI and GUI 206 or both GUIs may be displayed on the display screen 202 simultaneously. This allows the user 102 to view the maintenance instructions for the equipment corresponding to the 3D model 212 while viewing a specific portion of the 3D model 212 in the navigation window 210 of GUI 206.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of KOBEL, with wherein the building data further includes one or more of instructions, images, and/or videos, and wherein the at least one processor is further configured to: identify at least one or more of the instructions, images, and/or videos based upon the request; and cause the user computing device to display the identified one or more of the instructions, images, and/or videos., as provided by Schuster. The modification provides an improved system and method for generating augmented reality interfaces providing locations of components within a building, thereby to reduc[ing] required skill or expertise, minimizing labor time, optimizing use of materials, and other benefits in construction, manufacturing, and related industries. (Schuster, see par. [0025]).
Regarding claim 10. A computer-implemented method performed by an analytics computing device including at least one processor in communication with at least one memory device, the computer-implemented method that performs same steps of claim 1. Therefore, claim 10 is further rejected based on the same rationale as claim 1 set forth above and incorporated herein.
Regarding claims 11, The computer-implemented method of claim 11 performs the same step of claim 2. Therefore, claim 11 is further rejected based on the same rationale as claim 2 set forth above and incorporated herein.
Regarding claim 12. The computer-implemented method of claim 12, performs the same step of claim 3. Therefore, claim 12 is further rejected based on the same rationale as claim 3 set forth above and incorporated herein.
Regarding claim 13. The computer-implemented method of claim 13, performs the same step of claim 4. Therefore, claim 13 is further rejected based on the same rationale as claim 4 set forth above and incorporated herein.
Regarding claim 14. The computer-implemented method of claim 14, performs the same step of claim 5. Therefore, claim 15 is further rejected based on the same rationale as claim 5 set forth above and incorporated herein.
Regarding claim 15. The computer-implemented method of claim 15, performs the same steps of claim 6. Therefore, claim 16 is further rejected based on the same rationale as claim 6 set forth above and incorporated herein.
Regarding claim 19. At least one non-transitory computer-readable storage media having computer-executable instructions embodied thereon, wherein when executed by an analytics computing device comprising at least one processor in communication with at least one memory device, the computer-executable instructions cause the at least one processor (KOBEL, see par. [0117]) to perform same steps of claim 1. Therefore, claim 19 is further rejected based on the same rationale as claim 1 set forth above and incorporated herein.
Regarding claim 20. The at least one non-transitory computer-readable storage media of claim 20 performs same step of claim 2. Therefore, claim 20 is further rejected based on the same rationale as claim 1 set forth above and incorporated herein.
Claims 7, 9, 16 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over KOBEL et al. (US 20230196688 A1) in view of Sinha et al. (US 20240403614 A1), further in view of Schuster (US 20180336732 A1), as applied claim 1 above and further in view of Bedadala et al. (US 20180329993 A1).
Regarding claim 7. KOBEL in view of Sinha, and further in view of Schuster discloses the analytics computing device of claim 1 (as rejected above), but KOBEL in view of Sinha, and further in view of Schuster further does not disclose wherein the at least one processor is further configured to:
receive the request as a natural language input; and determine the inquiry from the natural language input using at least one chatbot executed by the at least one processor. However,
Bedadala discloses:
wherein the at least one processor is further configured to:
receive the request as a natural language input (Bedadala, see at least par. [0028] In contrast to the natural language processing and AI technology discussed above, the disclosed technology can receive a voice query in a natural language and translate it from natural language into a native database management language to respond to the query. For example, a human can ask his or her computer to “show large emails from December 2016,” and a data agent on the computer can receive the voice request, convert audio associated with the voice request to words in a natural language, convert that natural language into a domain specific language (e.g., SQL query), and convert the SQL query into a database management query.); and
determine the inquiry from the natural language input using at least one chatbot executed by the at least one processor (Bedadala, see at least par. [0327] Process 500 begins at block 505 and continues to block 510. Process 500 generally includes receiving text in a natural language, translating the text into a database query, and responding to the query in the chatbox with a response, answer, or graphic for the user. After receiving text from a user in an application or web browser that is running the chatbot, a computer or mobile device can implement process 500.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of KOBEL, with wherein the at least one processor is further configured to:
receive the request as a natural language input; and determine the inquiry from the natural language input using at least one chatbot executed by the at least one processor, as provided by Bedadala. The modification provides an improved system and method for generating augmented reality interfaces providing locations of components within a building, thereby to improve accuracy, a computer should access as many examples as possible of how custom words are used in the domain. The more sentences that represent the context in which speakers use words from the domain, the better the accuracy (Bedadala, see par. [0026]).
Regarding claim 9. KOBEL in view of Sinha, further in view of Schuster and further in view of Bedadala discloses the analytics computing device of claim 7 (as rejected above), and KOBEL in view of Sinha, further in view of Schuster and further in view of Bedadala further discloses wherein the user computing device is configured to execute a mobile application that causes the user computing device to at least capture the image stream and to display the user interface (KOBEL, see at least par. [0097] The application 1414 may be configured for inspecting, maintaining, troubleshooting and repairing the AR-compatible equipment 1406. In some embodiments, the AR troubleshooting module may overlay a 3D model or digital twin of the AR-compatible equipment 1406 over a live image of the real-world equipment 1406 on the screen of the user device. For example, the user 1402 may make a selection to enter an augmented reality portion of the application 1414. The user 1402 may then hold the user device so that a camera of the user device 1404 captures a live image of the equipment 1406 and the screen of the user device 1404 displays the live image.).
Regarding claim 16. A computer-implemented method of claim 16 performs same step of claim 7. Therefore, claim 16 is further rejected based on the same rationale as claim 7 set forth above and incorporated herein.
Regarding claim 18. A computer-implemented method of claim 18 performs same step of claim 9. Therefore, claim 18 is further rejected based on the same rationale as claim 9 set forth above and incorporated herein.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over KOBEL et al. (US 20230196688 A1) in view of Sinha et al. (US 20240403614 A1), further in view of Schuster (US 20180336732 A1), further in view of Bedadala et al. (US 20180329993 A1), as applied claim 7 above and further in view of KANG et al. (US 20210035576 A1).
Regarding claim 8. KOBEL in view of Sinha, and further in view of Schuster and further in view of Bedadala discloses the analytics computing device of claim 7 (as rejected above), but KOBEL in view of Sinha, further in view of Schuster and further in view of Bedadala does not disclose wherein the at least one processor is further configured to generate the image overlay based at least in part upon data output by the chatbot. However,
KANG discloses:
wherein the at least one processor is further configured to generate the image overlay based at least in part upon data output by the chatbot (KANG, see at least par. [0206] For example, referring to FIG. 17, the processor 120 may output a notification window 1702 through which information 1731 and 1732 relating to a content providable to the chatbot 1710 is readable, based on the pizza order intent of the user input 1701. The processor 120 may output a notification window 1733 that is of the type of a card including the information 1731 and 1732 relating to the content, and is superimposed on a screen 1730 (e.g. a chat room with the chatbot 1710), based on reception of a user input 1703 (e.g. tap, drag, swipe-up, etc.) relating to the notification window 1702. According to an embodiment, if there are multiple contents providable to the chatbot 1710, the notification window 1733 that is of the type of a card may display another content, based on a left-right drag (or swipe) input.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the system and method of KOBEL, with wherein the at least one processor is further configured to generate the image overlay based at least in part upon data output by the chatbot, as provided by KANG. The modification provides an improved system and method for generating augmented reality interfaces providing locations of components within a building, thereby to provide a user interface for allowing a user of an electronic device to chat with one or more artificial intelligence devices. In addition, while a user of an electronic device is chatting with one or more artificial intelligence devices through a single user interface, information that the user does not want to transfer may be transferred to an external device. (KANG, see par. [0003]).
Regarding claim 17. The computer-implemented method of claim 17, perform the same steps of claim 8. Therefore, claim 17 is further rejected based on the same rationale as claim 8 set forth above and incorporated herein.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KIM THANH THI TRAN whose telephone number is (571)270-1408. The examiner can normally be reached Monday-Friday 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ALICIA HARRINGTON can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KIM THANH T TRAN/Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
Prosecution Insights