DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
3. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
4. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
5. Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Seneviratne et al. (US 2024/0282068 A1) in view of Bayless et al. (US 2025/0111192 A1) and Paulson et al. (US 2024/0325088 A1).
6. With reference to claim 1, Seneviratne teaches A guided visual diagnosis method for equipment failures, (“the IAMIS is used to provide in-situ inspection of defects. To increase AFP production rates to match their potential, examples described herein include a visualization system for use with the IAMIS. The visualization system may be used to visualize one or more defects on a part identified by non-destructive testing. … FIG. 1 shows of an example defect visualization system 100 for use in visualizing one or more defects on a part 102 constructed by an automated fiber placement (AFP) system 104.” [0020-0021] “FIG. 7 shows an example method 700 of visualizing one or more defects on a part constructed by an AFP system.” [0042]) Seneviratne also teaches in response to receiving information of a failure associated with an equipment, using the diagnostic plan and real-time augmented reality (AR) indicators to guide a user through a diagnosis process, the AR configured to overlay visual indicators in visual data captured by the user; (“the IAMIS is used to provide in-situ inspection of defects. To increase AFP production rates to match their potential, examples described herein include a visualization system for use with the IAMIS. The visualization system may be used to visualize one or more defects on a part identified by non-destructive testing. In some examples, the visualization system includes an augmented reality (AR) viewing device and a visualization module that allows the defects to be visualized. The visualization system determines a position of the AR viewing device in relation to the part, before displaying each defect in virtual three-dimensional space (e.g., in an AR environment) according to their location in real three-dimensional space (e.g., in a real-world environment). In some examples, a scannable locator on the tool is scanned, and the position of the AR viewing device is determined in relation to the part. The AR viewing device may then retrieve defect data to display each defect in alignment with the location of the corresponding defect on the part.” [0020] “receiving defect data is initiated by scanning the scannable locator 124, which may be unique to each part 102. For example, the user may use the AR viewing device 110 to look closely at the scannable locator 124, and the defect data regarding that part 102 will be received by the visualization module 120.” [0023] “the x-axis of the scannable locator 124 is parallel to the x-axis of an AFP robot program before the robot is used for AFP layup or in-process inspection, and the y-axis of the scannable locator 124 coincides or overlaps with a laser line 202 from a profilometer, which is a part of the IAMIS defect detection system.” [0026] “The visualization module 120 then moves the generated scannable locator game object, existing in the AR environment, to match the location of the real-world scannable locator 124 so that they overlap.” [0028] “The display objects 300 may be overlaid onto the areas of real-world defects on the part 102. Alternatively, the detected defects may be projected onto a virtual part created within a virtual environment.” [0030] “The defect visualization system includes an AR viewing device 110 including a display 122, and a visualization module 120 configured to receive defect data representative of one or more defects on the part 102, determine a position of the AR viewing device 110 in relation to the part 102, generate one or more display objects 300 corresponding to the one or more defects on the part 102, and presenting the one or more display objects 300 on the display 122 so that the one or more display objects 300 appear in alignment with the one or more defects on the part 102.” [0047]) Seneviratne further teaches using an image-builder, which builds an image-based model to detect areas and parts for scene understanding, the image-builder using a perception module, to analyze and track a user action to enhance 3D scene understanding; (“once the scannable locator 124 is scanned, the visualization module 120 is configured to generate a scannable locator game object. As used herein, the term “game object” may refer to a building block within the structure or organization of the AFP robot program. For example, a game object may represent an entity or element within the AR environment that can be manipulated, interacted with, and/or rendered on the display 122.” [0027] “an AR touch menu 400 may be in view of the user as they make a touch selection to toggle through different types of defects. … The display objects 300 may be overlaid onto the areas of real-world defects on the part 102. Alternatively, the detected defects may be projected onto a virtual part created within a virtual environment. To generate a display object 300, the visualization module 120 may spawn a part model by adding a computer-aided-design (CAD) or other three-dimensional file (e.g., an STL file) to the IAMIS viewer application's persistent data storage. … the .txt file containing the detected defect information is in the IAMIS viewer application's persistent data storage. From this .txt file, defect type and defect contour data points are retrieved and a new SplitLine object is created using this data. The SplitLine object may define a line or boundary within the AR environment. For each SplitLine object, a new corresponding prefab game object is created. This time, the prefab game object includes a line renderer component. The defect contour data points are assigned to the line renderer from SplitLine, which draws the line in three-dimensional space. … the prefab game object has a child object containing a box collider, which is configured to detect each time a user touches the lines with their finger. The AR viewing device 110 is configured to be compatible with these touch interactions, which trigger the spawning of a text box that displays defect data retrieved from the text file. Once spawned, the text box and the displayed defect data are visible to the user of the AR viewing device 110.” [0030-0033] “the method includes initiating a virtual model game object based upon a virtual model of the part 102. The virtual model is loaded into the visualization module 120 either automatically or manually by a user. Once loaded, the scannable locator game object and the virtual model game object are correlated in virtual three-dimensional space. This correlation occurs by transforming the virtual model game object and the scannable locator game object, and then matching the transformed virtual game objects so that they are in the same position in virtual three-dimensional space. … the defect visualization system 100 may create one or more game objects (e.g., display object 300) for a virtual model of the part 102 and position the one or more game objects in the AR environment. The defect visualization system includes an AR viewing device 110 including a display 122, and a visualization module 120 configured to receive defect data representative of one or more defects on the part 102, determine a position of the AR viewing device 110 in relation to the part 102, generate one or more display objects 300 corresponding to the one or more defects on the part 102, and presenting the one or more display objects 300 on the display 122 so that the one or more display objects 300 appear in alignment with the one or more defects on the part 102.” [0046-0047]) Seneviratne teaches a 2D nested object detection model (“the prefab game object has a child object containing a box collider, which is configured to detect each time a user touches the lines with their finger. The AR viewing device 110 is configured to be compatible with these touch interactions, which trigger the spawning of a text box that displays defect data retrieved from the text file. Once spawned, the text box and the displayed defect data are visible to the user of the AR viewing device 110.” [0033])
PNG
media_image1.png
499
604
media_image1.png
Greyscale
Seneviratne does not explicitly teach using a large language model (LLM) to build a knowledge graph (KG) that is used to generate a diagnostic plan, wherein the KG is constructed by a KG builder that uses the LLM to extract information from one or more documents to construct a graph comprising at least one of a part, component, spatial area, or a diagnostic task related to a text source, thereby reducing a need for manual data labeling or model training; one or more perception sensors, generating an indicator database comprising overlay annotations extracted from at least one of 2D images, paths, markers, or messages; utilizing a 3D positioning to determine at least one of an object size, an orientation, a position, or a coverage of objects in a scene relative to a diagnostic checklist, and processing motion sensor data to ensure correct orientation and alignment during a diagnosis; and employing AR tracking and interaction modules to manage visual indicators and guide the user through the diagnostic checklist, until all checkpoints in the diagnostic checklist are satisfied. This is what Bayless teaches. Bayless teaches using a large language model (LLM) to build a knowledge graph (KG) that is used to generate a diagnostic plan, wherein the KG is constructed by a KG builder that uses the LLM to extract information from one or more documents to construct a graph comprising at least one of a part, component, spatial area, or a diagnostic task related to a text source, thereby reducing a need for manual data labeling or model training; (“The teacher model is trained on labeled data, and the student model is trained to mimic the teacher's behavior using unlabeled data of “soft targets”, which are probability distributions indicating the teacher's confidence in its predictions. By minimizing the difference between the student's predictions and the teacher's soft targets, the student model can learn from the teacher's knowledge and achieve similar or better performance, even with fewer parameters.” [0046] “FIG. 2 illustrates a component diagram 200 of example components of a service provider system 102 that includes knowledge-graph system 104 that generates knowledge graphs 122 using LLMs 118.” [0063] “the knowledge graphs 122 and curated knowledge graphs 202 may be graph-like data structures with nodes, edges, attributes, and labels.” [0065] “The query engine 204 may identify relevant nodes in the knowledge graphs 122/202 as answers to a given query. The query engine 204 (and/or the chatbot system 110) may utilize retrieval augmented generation (RAG) to determine the answer provided by the information stored in the knowledge graphs 122/202. RAG is an AI framework for retrieving facts from an external knowledge base to ground language models on the most accurate, up-to-date information and to give users insight into LLMs' generative process. … The query engine 204 and/or chatbot system 110 utilizes RAG to provide answers to queries and strikes a balance between the strengths of generative models (creative text generation) and retrieval-based systems (access to specific information).” [0067] “the prompt-engineering component 208 may provide the LLM 118 with a portion of the knowledge graph 122, such as the most relevant information for the formal-language query 128. For example, the query engine 204 may determine which portion of the knowledge graph 122 is semantically most relevant, or has answers that are relevant for the formal-language query 128. The prompt-engineering component 208 may then provide the most relevant information along with the formal-language query 128 to the LLM 118 in one or more prompts 132. Further, rather than providing the relevant information in the language or structure of the knowledge graph (e.g., semantic triples, RDF triples, etc.), the prompt-engineering component 208 may generate a summary (potentially harnessing an LLM) of the relevant information to reduce the amount of data provided to the LLM 118.” [0069] “the service provider system 102 may include internal data sources 218 (e.g., service documentation) usable to improve the knowledge graphs 122. The internal data sources 218 may include data provided by users 106, documentation generated by the service provider, and/or other information.” [0072] “Another alternative is that the provenance information can be provided in systems where explainability is key (e.g., a domain-specific medical AI assistant, where a physician wants to understand why the assistant may be recommending a particular diagnosis or course of treatment before finalizing a plan for the patient; a domain-specific legal AI assistant where an attorney needs to fact check its output in terms of the statutes and cases that influenced its output, etc.).” [0133]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Bayless into Seneviratne, in order to perform complex tasks on behalf of users.
The combination of Seneviratne and Bayless does not explicitly teach one or more perception sensors, generating an indicator database comprising overlay annotations extracted from at least one of 2D images, paths, markers, or messages; utilizing a 3D positioning to determine at least one of an object size, an orientation, a position, or a coverage of objects in a scene relative to a diagnostic checklist, and processing motion sensor data to ensure correct orientation and alignment during a diagnosis; and employing AR tracking and interaction modules to manage visual indicators and guide the user through the diagnostic checklist, until all checkpoints in the diagnostic checklist are satisfied. These are what Paulson teaches. Paulson teaches one or more perception sensors, (“A controller 216 can interface with the frame 202 to provide video input to the display(s) 206, receive output from the camera 210, ambient light sensor 212, and/or motion sensor(s) 214, and to provide power to the components in or otherwise coupled to the frame 202.” [0058]) Paulson also teaches generating an indicator database comprising overlay annotations extracted from at least one of 2D images, paths, markers, or messages; utilizing a 3D positioning to determine at least one of an object size, an orientation, a position, or a coverage of objects in a scene relative to a diagnostic checklist, and processing motion sensor data to ensure correct orientation and alignment during a diagnosis; and employing AR tracking and interaction modules to manage visual indicators and guide the user through the diagnostic checklist, until all checkpoints in the diagnostic checklist are satisfied. (“The augmented reality environment includes at least one visual guide indicating a separation distance between a reference location and the anatomical target. Generating the augmented reality environment includes overlaying the at least one visual guide with a view of the subject in a real-world environment.” [0002] “a computer system determines the position and orientation of one or more virtual objects to be displayed to a user in order to augment the real-world environment. … one or more markers (e.g., fiducial markers, radio frequency (“RF”) tracking coils) positioned in the real-world environment are identified, detected, or otherwise recognized in an image. A virtual object can then be generated and placed at, or in a location relative to, a detected marker. Additionally or alternatively, the virtual object can include one or more guides presented to the user and indicating positional information about the one or more markers.” [0016-0017] “one or more visual guides can be displayed in the virtual environment, or in an augmented reality environment. For instance, visual guides can indicate relative distances between two points or objects (e.g., a target location and a reference location). As a non-limiting example, the visual guides can include a first visual guide indicating a relative distance along a first spatial dimension and a second visual guide indicating a relative distance along a second spatial dimension that is preferably orthogonal to the first spatial dimension. In some implementations, a third visual guide can also be displayed, where the third visual guide indicates a relative distance along a third spatial dimension, which may preferably be orthogonal to both the first and second spatial dimensions. As an example, the visual guides can include guides that indicate a relative measure of anterior-posterior separation, right-left separation, and/or superior-inferior separation between a target location and a reference location.” [0024] “data collected by the virtual reality system used by the patient (e.g., the HMD, sensors embedded in or coupled to the HMD) can also be collected as used for post-processing of images or other data acquired from the patient.” [0041] “The radiation therapists would align the actual patient to the virtual model, confirming that any shifts are made in the correct directions. Additionally or alternatively, one or more visual guides can be displayed to a radiation therapist to indicate the relative positioning between a target location in the patient (e.g., a prescribed PTV) and a reference location (e.g., the treatment isocenter). These simplified visual guides can indicate when the target location is properly aligned with the reference location without the need to generating and displaying a more complex patient model. The visual guides may additionally or alternatively provide feedback on the alignment of the treatment isocenter with other target locations. For example, a first set of visual guides may indicate relative positions between a PTV and the treatment isocenter, whereas a second set of visual guides may indicate relative positions between one or more OARs and the treatment isocenter. In this way, the radiation therapist can be provided with feedback not only about whether the PTV is properly aligned with the treatment isocenter, but also whether one or more OARs will be in the radiation beam path.” [0043] “user interaction with the scene can be used as feedback for controlling other systems. For example, user interaction with the scene can provide feedback to a radiation treatment system, such that radiation is only delivered when the user (i.e., the patient) satisfies a criterion within the virtual/augmented reality environment (e.g., aligning a portion of their anatomy with a prescribed treatment contour).” [0050] “In the tracking block 402, magnetic field gradients are applied to sample spatial position (e.g., the x-, y-, and z-positions) of a tracking RF coil that is coupled to the medical device being used for the interventional procedure (e.g., a catheter). As a non-limiting example, the tracking RF coil can be a micro RF coil. In some embodiments, more than one tracking RF coils can be coupled to the medical device. A combination of zero-phase-reference and Hadamard encoding can be applied to correct for B.sub.0 inhomogeneities. Phase field dithering can be integrated to eliminate B.sub.1 inhomogeneities induced by the medical device.” [0078]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Paulson into the combination of Seneviratne and Bayless, in order to ensure that treatment is administered safely, correctly, and efficiently to a patient.
7. With reference to claim 2, Seneviratne does not explicitly teach the one or more documents comprise at least one of a manual or a text. This is what Bayless teaches (“The query engine 204 may integrate the retrieved information from the knowledge graphs 122/202 to provide context or augment the generative model's responses. The integration can be done in various ways, such as concatenating the retrieved information with the input prompt, using it as a context window, or employing attention mechanisms to focus on specific parts of the retrieved content. The query engine 204 may then generate output using combined input (original prompt+retrieved information) that is passed to the generative model, and the model then generates a response based on this augmented input. The query engine 204 and/or chatbot system 110 utilizes RAG to provide answers to queries and strikes a balance between the strengths of generative models (creative text generation) and retrieval-based systems (access to specific information). By combining these components, the query engine 204 and/or chatbot system 110 aims to produce responses that are both contextually relevant and factually accurate.” [0067] “the service provider system 102 may include internal data sources 218 (e.g., service documentation) usable to improve the knowledge graphs 122. The internal data sources 218 may include data provided by users 106, documentation generated by the service provider, and/or other information.” [0072]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Bayless into Seneviratne, in order to perform complex tasks on behalf of users.
8. With reference to claim 3, Seneviratne teaches the 2D nested object detection model (“the prefab game object has a child object containing a box collider, which is configured to detect each time a user touches the lines with their finger. The AR viewing device 110 is configured to be compatible with these touch interactions, which trigger the spawning of a text box that displays defect data retrieved from the text file. Once spawned, the text box and the displayed defect data are visible to the user of the AR viewing device 110.” [0033])
Seneviratne does not explicitly teach uses a training dataset comprising a relatively small set of training samples to increase a detection accuracy. This is what Bayless teaches (“While large models may have state-of-the-art performance, in various scenarios it may be desirable to deploy a smaller model. Knowledge distillation is a technique that transfers knowledge from a complex neural network (the “teacher model”) to a simpler one (the “student model”). The teacher model is trained on labeled data, and the student model is trained to mimic the teacher's behavior using unlabeled data of “soft targets”, which are probability distributions indicating the teacher's confidence in its predictions. By minimizing the difference between the student's predictions and the teacher's soft targets, the student model can learn from the teacher's knowledge and achieve similar or better performance, even with fewer parameters.” [0046]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Bayless into Seneviratne, in order to perform complex tasks on behalf of users.
9. With reference to claim 4, the combination of Seneviratne and Bayless does not explicitly teach the one or more perception module provides feedback and a recommendation based on the analysis and tracking of actions. This is what Paulson teaches (“As the patient is breathing, the visual guides can be updated in real-time to indicate how respiration is affecting the positioning of the target location relative to the reference location. In this way, the user can be provided with feedback on whether, and how, to control breathing to better align the target and reference locations.” [0036] “real-time information that can be displayed to the user can include data that provide a visual feedback to the patient regarding the imaging scan being conducted or to provide an awareness of the patient's motion. … data collected by the virtual reality system used by the patient (e.g., the HMD, sensors embedded in or coupled to the HMD) can also be collected as used for post-processing of images or other data acquired from the patient.” [0040-0041] “The radiation therapists would align the actual patient to the virtual model, confirming that any shifts are made in the correct directions. Additionally or alternatively, one or more visual guides can be displayed to a radiation therapist to indicate the relative positioning between a target location in the patient (e.g., a prescribed PTV) and a reference location (e.g., the treatment isocenter). These simplified visual guides can indicate when the target location is properly aligned with the reference location without the need to generating and displaying a more complex patient model. The visual guides may additionally or alternatively provide feedback on the alignment of the treatment isocenter with other target locations. For example, a first set of visual guides may indicate relative positions between a PTV and the treatment isocenter, whereas a second set of visual guides may indicate relative positions between one or more OARs and the treatment isocenter. In this way, the radiation therapist can be provided with feedback not only about whether the PTV is properly aligned with the treatment isocenter, but also whether one or more OARs will be in the radiation beam path.” [0043] “user interaction with the scene can provide feedback to a radiation treatment system, such that radiation is only delivered when the user (i.e., the patient) satisfies a criterion within the virtual/augmented reality environment (e.g., aligning a portion of their anatomy with a prescribed treatment contour).” [0050] “A controller 216 can interface with the frame 202 to provide video input to the display(s) 206, receive output from the camera 210, ambient light sensor 212, and/or motion sensor(s) 214, and to provide power to the components in or otherwise coupled to the frame 202.” [0058]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Paulson into the combination of Seneviratne and Bayless, in order to ensure that treatment is administered safely, correctly, and efficiently to a patient.
10. With reference to claim 5, the combination of Seneviratne and Bayless does not explicitly teach at least one of the feedback or the recommendation is provided in real-time. This is what Paulson teaches (“As the patient is breathing, the visual guides can be updated in real-time to indicate how respiration is affecting the positioning of the target location relative to the reference location. In this way, the user can be provided with feedback on whether, and how, to control breathing to better align the target and reference locations.” [0036] “real-time information that can be displayed to the user can include data that provide a visual feedback to the patient regarding the imaging scan being conducted or to provide an awareness of the patient's motion. … data collected by the virtual reality system used by the patient (e.g., the HMD, sensors embedded in or coupled to the HMD) can also be collected as used for post-processing of images or other data acquired from the patient.” [0040-0041] “The radiation therapists would align the actual patient to the virtual model, confirming that any shifts are made in the correct directions. Additionally or alternatively, one or more visual guides can be displayed to a radiation therapist to indicate the relative positioning between a target location in the patient (e.g., a prescribed PTV) and a reference location (e.g., the treatment isocenter). These simplified visual guides can indicate when the target location is properly aligned with the reference location without the need to generating and displaying a more complex patient model. The visual guides may additionally or alternatively provide feedback on the alignment of the treatment isocenter with other target locations. For example, a first set of visual guides may indicate relative positions between a PTV and the treatment isocenter, whereas a second set of visual guides may indicate relative positions between one or more OARs and the treatment isocenter. In this way, the radiation therapist can be provided with feedback not only about whether the PTV is properly aligned with the treatment isocenter, but also whether one or more OARs will be in the radiation beam path.” [0043]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Paulson into the combination of Seneviratne and Bayless, in order to ensure that treatment is administered safely, correctly, and efficiently to a patient.
11. With reference to claim 6, Seneviratne does not explicitly teach t the text source comprises at least one of a parts lists or a failure report. This is what Bayless teaches (“The query engine 204 may integrate the retrieved information from the knowledge graphs 122/202 to provide context or augment the generative model's responses. The integration can be done in various ways, such as concatenating the retrieved information with the input prompt, using it as a context window, or employing attention mechanisms to focus on specific parts of the retrieved content. The query engine 204 may then generate output using combined input (original prompt+retrieved information) that is passed to the generative model, and the model then generates a response based on this augmented input. The query engine 204 and/or chatbot system 110 utilizes RAG to provide answers to queries and strikes a balance between the strengths of generative models (creative text generation) and retrieval-based systems (access to specific information). By combining these components, the query engine 204 and/or chatbot system 110 aims to produce responses that are both contextually relevant and factually accurate.” [0067] “he prompt 132 may include a request that the LLM 118 generate a truthful answer for the query 128, may include the natural-language query 126, and may further include one or more triple patterns (e.g., a list of triples where some elements are variables). The prompt-engineering component 208 may then generate the prompt 132 that consists of this particular triple pattern and instructions about how to evaluate it. … The LLM 118 may then response with a list of assignments 302 to each variable in the triple pattern.” [0085-0086] “the chatbot UI 124 may include or expose a feedback mechanism, such as a report-highlighted-errors option 410, through which users 106 are able to provide feedback regarding the accuracy of an answer provided by the chatbot.” [0089]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Bayless into Seneviratne, in order to perform complex tasks on behalf of users.
12. With reference to claim 7, the combination of Seneviratne and Bayless does not explicitly teach the one or more perception sensors comprise a camera and/or motion sensor. This is what Paulson teaches (“controller 216 can interface with the frame 202 to provide video input to the display(s) 206, receive output from the camera 210, ambient light sensor 212, and/or motion sensor(s) 214, and to provide power to the components in or otherwise coupled to the frame 202.” [0058]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Paulson into the combination of Seneviratne and Bayless, in order to ensure that treatment is administered safely, correctly, and efficiently to a patient.
13. With reference to claim 8, Seneviratne teaches performing an initial calibration comprises mapping the user and the equipment in a 3D space by using 2D images and AR tracking. (“the AR viewing device 110 projects or presents one or more defects using a display 122 such that the user can view the defects on the real-world part 102 as the user physically moves around the real-world part 102. … receiving defect data is initiated by scanning the scannable locator 124, which may be unique to each part 102. For example, the user may use the AR viewing device 110 to look closely at the scannable locator 124, and the defect data regarding that part 102 will be received by the visualization module 120.” [0022-0023] “The position of the AR viewing device 110 in relation to the part 102 is determined at operation 720. The scannable locator 124 may be used as a location reference to determine the position of the AR viewing device 110 in relation to the part 102. … When this occurs, the scannable locator game object location may be matched with the location of the scannable locator 124 on the tool 106. The scannable locator 124 may also be used as an origin by the AR viewing device 110 to map detected defects.” [0043] “A display object 300 for each of the one or more defects is generated at operation 740. The display objects 300 may be presented on a display 122 of the AR viewing device 110 so that the display object 300 appears in alignment with the location of the actual defect on the part 102. In some examples, the method includes initiating a virtual model game object based upon a virtual model of the part 102. The virtual model is loaded into the visualization module 120 either automatically or manually by a user. Once loaded, the scannable locator game object and the virtual model game object are correlated in virtual three-dimensional space. This correlation occurs by transforming the virtual model game object and the scannable locator game object, and then matching the transformed virtual game objects so that they are in the same position in virtual three-dimensional space.” [0045-0046])
14. With reference to claim 9, Seneviratne teaches each of the 2D images comprises at least a portion of an object of interest. (“once the scannable locator 124 is scanned, the visualization module 120 is configured to generate a scannable locator game object. As used herein, the term “game object” may refer to a building block within the structure or organization of the AFP robot program. For example, a game object may represent an entity or element within the AR environment that can be manipulated, interacted with, and/or rendered on the display 122. Game objects may encompass a wide range of entities or elements, including characters, items, obstacles, textures, and/or special effects.” [0027] “FIGS. 3 and 4 show views of a user as they view parts 102 with defects through the AR viewing device 110. The visualization module 120 generates a display object 300 for each of the defects and communicates with the AR viewing device 110 to present or display the display object 300 for each of the defects using the display 122. In this manner, the AR viewing device may allow a user to visualize at least one of a number of defects created or caused by the AFP system 104. … a first color (e.g., red) may be used to visualize a first type of defect (e.g., a gap), and a second color (e.g., green) may be used to visualize a second type of defect (e.g., a missing tow). The display objects 300 may be overlaid onto the areas of real-world defects on the part 102. Alternatively, the detected defects may be projected onto a virtual part created within a virtual environment.” [0029-0030])
15. Claim 10 is similar in scope to claim 1, and thus is rejected under similar rationale. Seneviratne additionally teaches A non-transitory computer-readable medium for storing instructions for executing a process, (“Embodiments of the aspects of the present disclosure may be described in the general context of data and/or processor-executable instructions, such as program modules, stored one or more tangible, non-transitory storage media and executed by one or more processors or other devices. … The processor-executable instructions may be organized into one or more processor-executable components or modules on a tangible processor readable storage medium.” [0052-0053])
16. Claims 11-18 are similar in scope to claims 2-9, and they are rejected under similar rationale.
17. Claim 19 is similar in scope to claim 1, and thus is rejected under similar rationale. Seneviratne additionally teaches An apparatus, comprising: a processor, (“Examples of computing systems, environments, and/or configurations that may be suitable for use with aspects of the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. Embodiments of the aspects of the present disclosure may be described in the general context of data and/or processor-executable instructions, such as program modules, stored one or more tangible, non-transitory storage media and executed by one or more processors or other devices.” [0051-0052])
18. Claim 20 is similar in scope to claim 4, and thus is rejected under similar rationale.
Conclusion
19. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michelle Chin whose telephone number is (571)270-3697. The examiner can normally be reached on Monday-Friday 8:00 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/Awww.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kent Chang can be reached on (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https:/Awww.uspto.gov/patents/apply/patent- center for more information about Patent Center and https:/Awww.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE CHIN/
Primary Examiner, Art Unit 2614
Prosecution Insights