DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 16 is objected to because of the following informalities: the claim preamble reciting “a method for inspecting, the method implemented by an inspection computing device comprising at least one processor in communication with at least one memory device, wherein the process comprises:” should recite “a method for inspecting, the method implemented by an inspection computing device comprising at least one processor in communication with at least one memory device, wherein the method comprises:” for proper antecedent basis. Appropriate correction is required.
Response to Arguments
Applicant's arguments filed 09/03/2025, regarding the rejection of claims 1-10, 12-16, and 20-21, under 35 U.S.C. 103, have been fully considered but they are not persuasive. Applicant argues, on page 10 of the remarks, that “
PNG
media_image1.png
224
646
media_image1.png
Greyscale
”.
Examiner disagrees. Guo teaches training, via machine learning, an inspection model, an inspection model based on a plurality of training images, each of the training images associated with one of the stored plurality of classification codes (Guo, para. [0081]-[0083]; FIG. 4: “FIG. 4 is a block diagram illustrating a high-level overview of the training method 400 for the first machine learning model 106 to recognize a step in a product assembly process. At 402, the training method 400 receives the training dataset, the dataset having collections of two-dimensional images and three-dimensional models related to a step in a product assembly process. At 404, the training method 400 augments at least one two-dimensional image with data from the three-dimensional model. The augmented data template at least comprises the step in the process the image represents, the edge map of at least one base component, the edge map of at least one constituent component, the spatial relationship between them, and an acceptable tolerance between the ideal position of the constituent components and where it is placed by the worker. The tolerance may be measured using a percentage of area occupied, a boundary of pixels, or any other appropriate measure. At 406, the training method 400 stores the augmented image data in the template storage 110. For example, the training method 400 may fully populate the template storage 110 with at least one augmented image file for each step in the product assembly process. At S408, the first machine learning model 106 accesses the template storage 110 to use as references when comparing images captured by the image capture device 104.”; the term “template” used in Guo is equivalent to the claim term “classification code”; the term “classification code”, using broadest reasonable interpretation, means computer data indicating a specific state of a process; the augmented data template in Guo indicates the specific step of a process and is stored to train a machine learning model).
PNG
media_image2.png
730
686
media_image2.png
Greyscale
).
Further, Guo teaches to further train, via machine learning, the trained inspection model based on the current image and the classification code output by the trained inspection model based on the input of the current image (Guo, para. [0079]; para. [0062]: “The machine-learning algorithm 210 may be operated in a learning model using the training dataset 212 as input. The machine-learning algorithm 210 may be executed over a number of iterations using the data from the training dataset 212. With each iteration, the machine-learning algorithm 210 may update internal weighting factors based on the achieved results. For example, the machine-learning algorithm 210 may compare output results (e.g., annotations) with the results included in the training dataset 212. Since the training dataset 212 includes the expected results, the machine-learning algorithm 210 may determine when performance is acceptable”; “The computing device 102 stores, using the feature vector map storage 118, templates generated based on a combination of the data from at least one two-dimensional image and at least one three-dimensional model related to a step in a product assembly process. Computing device 102 may further generate additional training data sets conversions of input images for processing by the first machine learning model 106 and store them in the feature vector map storage 118. Feature vector maps are accurate by greater than a threshold (e.g., greater than or equal to 60% or other suitable percentage, value, absolute value, integer and the like) are stored with their corresponding input images as training data sets corresponding to the step they illustrate by the training dataset identifier. If the resulting similarity score is less than the predetermined threshold, the vector maps and their corresponding input images are discarded.”; the model gives feedback to the user visually with instructions to the user if the placement of an object in a product assembly process is incorrect when the worker is assembling the product; the continuously updated visual indication to the worker is part of the trained inspection model; therefore the re-training after each iteration is based on the current image in worker’s visual field, the feedback shown to the user, and the classification codes or “template” that allow model to know visually what specific step of the product assembly process the worker is at).
Therefore, the rejection of the claims under 35 U.S.C. 103 is maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-10, 12, 14-16, 20-21, and 23-34 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No.: 2019/0057548 (Singh et al.) (hereinafter Singh), in view of U.S. Patent Application Publication No.: 2023/0245448 (Guo et al.) (hereinafter Guo).
Regarding claim 1, Singh teaches a wearable inspection unit comprising: (Singh, para. [0022]: “FIG. 1 illustrates an augmented reality system 100 in accordance with an example embodiment. In this example, a user 10 performs operations on one or more types of industrial assets 130 which may include machine and equipment in the fields of transportation, energy, healthcare, manufacturing, and the like. Referring to FIG. 1, the system 100 includes an augmented reality (AR) server 110 in communication with an AR device 120 associated with the user 10. The AR server 110 may be a cloud platform, a server, or another computing device attached to a network. The AR device 120 may be one or more of glasses, a helmet, a screen, a camera, a microphone, and/or the like, which are associated with the user 10. In some examples, the AR device 120 or a plurality of AR devices may be attached to or worn by the user 10. As another example, the AR device 120 may be within a field of view of the user 10 but not attached to the user. The AR server 110 and the AR device 120 may be connected to each other by a network such as the Internet, private network, or the like. As another example, the AR device 120 may be connected to the AR server 110 by a cable or the AR device 120 may incorporate the features of the AR server 110 within the AR device 120.”;
PNG
media_image3.png
515
736
media_image3.png
Greyscale
);
at least one sensor (Singh, para. [0023]; see FIG. 1 above: “The AR device 120 may be outfitted with one or more data gathering components (e.g., cameras, sensors, LIDAR, thermal cameras, etc.) which are capable of capturing images, spatial data, audio, temperature, and the like, and which are configured to monitor respective operations or conditions of the user 10 performing operations with respect to an asset 130. Data captured by the AR device 120 can be recorded and/or transmitted to the AR server 120 or other remote computing environment described herein. By bringing the data into the AR system 100, the AR platform described herein which may include software or a combination of hardware and software may analyze a process being performed by the user 10 with respect to the asset 130 and provide augmented reality components that are related to the process. The AR software may be included in the AR server 110, the AR device 120, or a combination thereof. As a non-limiting example, the user 10 may be performing a maintenance process, a repair process, a cleaning process, a production/assembly process, or any other process known in which a user interacts with machines or equipment in an industrial setting. The AR server 120 may analyze the captured data and determine a current state of the process being performed by the user. Furthermore, the AR server 110 can provide augmented reality components to the AR device 120 based on a future state of the process being performed by the user 10. For example, the augmented reality components can indicate a process path or a next part in the operation that is to be replaced/inspected.”);
display component (Singh, para. [0027]: “FIG. 2 illustrates an augmented reality process 200 in accordance with an example embodiment. In this example, the augmented reality process 200 includes a plurality of components including an AR device 210 that captures process data and provides the process data to an object recognition module 220. The object recognition module performs object recognition from the data and provides the object recognized data to a process learning module 230. The process learning module determines a state of a manual industrial process 250 (or operation) and provides data about the state to a scene construction module 240. The scene construction module 240 generates AR components for display by the AR device 210 based on a scene in which a user/operator is performing the process 250. Here, the scene construction module may overlay holographic components within a field of view of the user/operator wearing the AR device 210 and feedback the AR components to the AR device 210. FIG. 2 also illustrates that the manual industrial process 250 performed by the user/operator includes a plurality of steps.”;
PNG
media_image4.png
405
796
media_image4.png
Greyscale
); and
a controller comprising at least one processor in communication with at least one memory device, and wherein the controller is in communication with the at least one sensor and the display component, wherein the at least one processor is programmed to: (Singh, para. [0044]: “FIG. 5 illustrates a computing system 500 for generating an augmented reality in accordance with an example embodiment. For example, the computing system 500 may be a cloud platform, a server, a user device, or some other computing device with a processor. Also, the computing system 500 may perform the method of FIG. 4. Referring to FIG. 5, the computing system 500 includes a network interface 510, a processor 520, an output 530, and a storage device 540. Although not shown in FIG. 5, the computing system 500 may include other components such as a display, an input unit, a receiver/transmitter, and the like. The network interface 510 may transmit and receive data over a network such as the Internet, a private network, a public network, and the like. The network interface 510 may be a wireless interface, a wired interface, or a combination thereof. The processor 520 may include one or more processing devices each including one or more processing cores. In some examples, the processor 520 is a multicore processor or a plurality of multicore processors. Also, the processor 520 may be fixed or it may be reconfigurable. The output 530 may output data to an embedded display of the device 500, an externally connected display, an AR device, a cloud instance, another device or software, and the like. The storage device 540 is not limited to any particular storage device and may include any known memory device such as RAM, ROM, hard disk, and the like.”;
PNG
media_image5.png
395
422
media_image5.png
Greyscale
);
train, via machine learning, an inspection model based on a plurality of training images (Singh, para. [0032]; para. [0035]: “According to various embodiments, the AR device 210 can be configured to capture and annotate data received from one or more AR devices 210 (such as images, audio, spatial data, temperature, etc.) which may be used by the process learning module 230 to train one or more machine learning models on how to complete the manual industrial operation. The training can be continually performed as data continues to be received from the AR device 210. Accordingly, the learning can be adaptive and dynamic based on a current user manual industrial operation and previous manual industrial operations. Furthermore, the scene construction module 240 may output the one or more AR components (i.e., scene components) based on the trained machine learning models.”; “The process was modeled using a recurrent neural network (RNN) that consumes the string encoded graph of the assembly process. The RNN was trained on a set of simulated data for the pick and place task and can predict the subsequent component given the current state. For example, if the RNN were trained on data and given a current state (component A), it would predict an equal likelihood that the next component to be operated on by the user in the operation is component B or component C. The system is trained to not only predict a process sequence of a manual industrial operation, but also suggest paths that are better quality, or more efficient. In the simulated data, some paths are more efficient and a RNN is trained to provide such paths. Similarly, other paths lead to higher quality and a separate RNN may be trained to provide high quality paths. Thus, the process learning module 230 can suggest paths that are more likely to proceed efficiently and/or with highest quality.);
receive a signal from the at least one sensor including a current image in the current view of a user (Singh, para. [0039]; FIG. 4: “FIG. 4 illustrates a method 400 for generating an augmented reality in accordance with an example embodiment. For example, the method 400 may be controlled by AR software executing on an AR device, an AR server, a cloud computing system, a user computing device, or a combination thereof. The software may control the hardware of the device to perform the method 400. Referring to FIG. 4, in 410, the method includes receiving data that is captured of a manual industrial operation being performed by a user. The manual industrial operations may be manufacturing of a component, repair, assembly, inspection, cleaning maintenance, and the like, performed by the user. The received data may include images, pictures, video, spatial data (spatial map), temperature, thermal data, and the like, captured of a user performing or about to perform the manual industrial operation. In some embodiments, the received data may also or instead include audio data such as spoken commands, instructions, dialogue, explanations, and/or the like. The image data may include a picture of a scene and/or a surrounding location at which the manual industrial operation is being performed, a picture of a machine or equipment, a picture of the user interacting with the machine or equipment or preparing to interact with the machine or equipment, and the like. The image data may be captured by an AR device such as a pair of glasses, a helmet, a band, a camera, and the like, which may be worn by or attached to the user.”;
PNG
media_image6.png
414
418
media_image6.png
Greyscale
);
input the current image to the trained inspection model to determine a classification code of a plurality of classification codes; store a plurality of states of a process (Singh, para. [0024]-[0025]: “Furthermore, the AR software may include a learning system. In this case, the learning system may receive a continuous stream or an intermittent stream of data from the AR device 120, and insights gained through analysis of such data can lead to enhancement of the process being performed by the user 10 based on asset designs, enhanced software algorithms for operating the same or similar assets, better operator efficiency, the current user 10 and/or other users previously performing similar process operations, and the like.” … According to various embodiments, the AR server 110 can analyze the images and/or audio coming in and determine a current state of the process being performed by the user 10 based on the analyzed images/audio with respect to a one or more models maintained by the AR server 110. For example, the AR server 110 may maintain a process map including images of the process performed previously by the user 10 or other users as well as descriptions, images, and audio of the individual steps/phases of the process being performed by the user 10.”; the process map includes descriptions, images, and audio of the steps of the process; this process map meets the broadest reasonable definition of a “classification code” that indicates a certain step of a process after comparing the input image with the trained machine learning model);
determine a current step of a process being performed by the user based on the classification code (Singh, para. [0040]; FIG. 4; para. [0038]; FIG. 3: “In 420, the method includes identifying a current state of the manual industrial operation that is being performed by the user based on the received image data. For example, the manual industrial operation may include a plurality of steps which are to be performed by the user including an initial step, a finishing step, and one or more intermediate steps. The AR software may identity a current step being performed by the user as the current state of the manual industrial operation. For example, the AR device executing the AR software may store a process map or model that includes reference pictures, images, description, sounds, etc., about each step of the manual industrial operation which are received from historical performances and/or the current performance of the manual industrial operation. The AR software may determine that the current step is the initial step, an intermediate step, the final step, and the like.”;
PNG
media_image7.png
414
418
media_image7.png
Greyscale
“For example, the scene construction module 240 may combine the process predictions from the process learning module 230 with business specific logic to generate scene components for display by the AR device 210. Examples may include, but are not limited to, simple holographic indicators, text displays, audio/video clips, images, etc. Location and placement of virtual objects in the scene are tracked in this module and updated based on the results of the process learning module. Results from this module are then transmitted to the AR device for display to the user in real-time. A non-limiting example of the scene construction 300 with AR components is shown in FIG. 3. For example, one or more objects may be recognized and shown as being completed within the process, currently being worked on within the process, and expected to be worked on at some point in the future. In this example, labels 310 are used to indicate components of a manual industrial process that have been completed by user 10 wearing AR device 110, while label 320 indicates a component of a current state (e.g., a current step) of the manual industrial process operation. Also, indicator 330 provides an indication of a position of the next or future state of the manufacturing process within the scene. This is just merely an example, and different displays, indicators, paths, etc., may be used to guide the user or enhance the user's understanding of the process.”
PNG
media_image8.png
540
833
media_image8.png
Greyscale
);
determine one or more feedback on the process being performed by the user based upon the current image and the classification code (Singh, para. [0017]: “The example embodiments provide an augmented reality (AR) platform that includes a learning system for human (or robot operated) manual industrial operations or processes such as manufacturing operations, repair operations, assembly, maintenance, inspection, and the like, especially in industrial settings such as manufacturing. The operations may be performed on machine, equipment, products, and the like, at a manufacturing plant or other environment, and may be a process that includes a plurality of stages, steps, phases, etc. The platform allows AR devices (e.g., eyeglasses, lenses, head gear, helmets, sensors, cameras, microphone, etc.) to capture real-time video and audio of the process being performed by the user which can be input to the learning system. The learning system may be coupled to the AR device or connected to the AR device via a network or cable. From the observed data, the learning system may generate and continuously update a process map of the operation being performed by the user that represents a current state of the operation and also can be used to predict a future state of the operation. The process map may be used to generate intuitive and efficient instructions for both novice and expert operators to aid and navigate the operator through the process. These instructions may also be delivered through the same AR device that captures the data. Thus, the AR device serves both as the data capture device for input to the learning system and as the content delivery device for the instructions generated by the learning system.”); and
provide a notification message to the user via augmented reality overlay based on the current step of the process being performed by the user, wherein the notification includes the one or more feedback on the process (Singh, para. [0042]-[0043]; FIG. 4: “Furthermore, in 440 the method includes outputting the one or more AR components to an AR device of the user for display based on a scene of the manual industrial operation. In some embodiments, the AR components may be output for display by the same AR device that captured the initial data of the operation being performed. For example, the image data may be captured by a pair of lenses and/or a helmet worn by the user, and the AR components may also be output to the pair of lenses and/or the helmet. In some embodiments, additional image data of the manual industrial operation being performed by the user is simultaneously received from the AR device being worn by the user while the one or more AR components are being output to the AR device being worn by the user. For example, the AR device may capture image data of a next step of the manual industrial operation being performed while the AR software outputs AR components of the next step of the manual industrial operation being performed. In some embodiments, the output AR components output in 440 may indicate a suggested path for performing the manual industrial operation within a field of view of the user. In some cases, holographic indicators may be output that include at least one of images, text, video, 3D objects, CAD objects, arrows, pointers, symbols, and the like, within the scene which can aid the user. Also, the AR software may update the AR components being output for display in the scene based on a progress of the manual industrial operation being performed by the user. For example, when the AR software detects that the user is performing the next step of the operation, the AR software may output AR components related to the step that is in the future with respect to the next step.”;
PNG
media_image9.png
414
418
media_image9.png
Greyscale
).
Singh fails to teach
each of the plurality of training images associated with one of the stores classification codes; wherein the classification code is output by the trained inspection model based on the input of the current image; and further train, via machine learning, the trained inspection model based on the current image and the classification code output by the trained inspection model based on the input of the current image.
Guo teaches
each of the plurality of training images associated with one of the stores classification codes (Guo, para. [0081]-[0083]; FIG. 4; para. [0080]; FIG. 3A-3B: “FIG. 4 is a block diagram illustrating a high-level overview of the training method 400 for the first machine learning model 106 to recognize a step in a product assembly process. At 402, the training method 400 receives the training dataset, the dataset having collections of two-dimensional images and three-dimensional models related to a step in a product assembly process. At 404, the training method 400 augments at least one two-dimensional image with data from the three-dimensional model. The augmented data template at least comprises the step in the process the image represents, the edge map of at least one base component, the edge map of at least one constituent component, the spatial relationship between them, and an acceptable tolerance between the ideal position of the constituent components and where it is placed by the worker. The tolerance may be measured using a percentage of area occupied, a boundary of pixels, or any other appropriate measure. At 406, the training method 400 stores the augmented image data in the template storage 110. For example, the training method 400 may fully populate the template storage 110 with at least one augmented image file for each step in the product assembly process. At S408, the first machine learning model 106 accesses the template storage 110 to use as references when comparing images captured by the image capture device 104.”; the term “template” used in Guo is equivalent to the claim term “classification code”; the term “classification code”, using broadest reasonable interpretation, means computer data indicating a specific state of a process; the augmented data template in Guo indicates the specific step of a process and is stored to train a machine learning model”;
PNG
media_image2.png
730
686
media_image2.png
Greyscale
“FIG. 3A illustrates an image captured by the image capture device from an overhead perspective relative to the position of the worker. The image in FIG. 3A is of a washing machine door that is yet to be assembled from the overhead perspective of the worker's station … FIG. 3B illustrates a subsequent display of the washing machine door with an overlay indicating the correct position for a constituent component to be installed, based on the template related to this step in the product assembly process, as instructed by AR assistant 114. Here, the component in FIG. 3A is the base component and the part to be installed is the constituent component. The spatial relationship is determined first by identifying the edges of the washing machine door, matching the edge map of the live image with the template related to this step. When the template and the live image are aligned, spatial data in the template indicate the correct position of the part to be installed in relation to the edges of the washing machine door. That spatial data is used to display an indication to the worker regarding the installation of the constituent component. The color of the indication may change to indicate correct placement, incorrect placement, or awaiting placement.”;
PNG
media_image10.png
494
654
media_image10.png
Greyscale
);
wherein the classification code is output by the trained inspection model based on the input of the current image (Guo, para. [0084]-[0086]; FIG. 5: “FIG. 5 provides a block diagram illustrating the method 500, a high-level overview for detecting a step in a product assembly process. At 502, the method 500 receives a real-time image from the overhead capture device 104. The image capture device 104 captures a continuous stream of real-time images; at least one of the images is sent to the first- and second-machine learning models. At 504, the method 500 receives the captured image. At 506, the method 500 received image may be converted into an oriented edge map via holistically-nested edge detection. That is followed by edge-based non-maximum suppression and edge orientation computation. At 508, the method 500 queries the resulting edge map against all the available templates in the template storage 110. The first machine learning model extracts the oriented edges from the edge map and the template and compares the 0 value at each (X, Y) coordinate of every pixel of the detected edges. The extracted oriented edges compare with each edge of the template. The template that most closely matches the captured image is determined; the matched template is designated as the initial prediction of the systems and methods described herein. At 510, the method 500 also sends the captured image to the second machine learning model 108, which converts the image into an edge map using deep learning (DL) edge detection. For example, one possible DL edge detector is canny edge detection. At 512, the method 500 the outputs an edge map generated by the second machine learning model which detects the edges of the captured image. At 514, the method 500 instructs the second machine learning model to use metric learning to compare the generated edge map with the initial prediction template. Each edge pixel is compared to a corresponding pixel from the captured image data. Comparisons of the 0 value at each (X, Y) coordinate generates, by the processor, a similarity value. If edge support exists at a ratio greater than a threshold (e.g., greater than or equal to 60% or other suitable percentage, value, absolute value, integer and the like) of all edges of any one template, that template indicates the prediction of the current step in the product assembly process is correct. Based on whether the initial prediction was verified, the method 500 continues to either 518 (if the prediction was successfully verified) or 520 (if the prediction was not successfully verified).”; the template is the “classification code”;
PNG
media_image11.png
1040
684
media_image11.png
Greyscale
); and
further train, via machine learning, the trained inspection model based on the current image and the classification code output by the trained inspection model based on the input of the current image (Guo, para. [0079]; para. [0062]: “The machine-learning algorithm 210 may be operated in a learning model using the training dataset 212 as input. The machine-learning algorithm 210 may be executed over a number of iterations using the data from the training dataset 212. With each iteration, the machine-learning algorithm 210 may update internal weighting factors based on the achieved results. For example, the machine-learning algorithm 210 may compare output results (e.g., annotations) with the results included in the training dataset 212. Since the training dataset 212 includes the expected results, the machine-learning algorithm 210 may determine when performance is acceptable”; “The computing device 102 stores, using the feature vector map storage 118, templates generated based on a combination of the data from at least one two-dimensional image and at least one three-dimensional model related to a step in a product assembly process. Computing device 102 may further generate additional training data sets conversions of input images for processing by the first machine learning model 106 and store them in the feature vector map storage 118. Feature vector maps are accurate by greater than a threshold (e.g., greater than or equal to 60% or other suitable percentage, value, absolute value, integer and the like) are stored with their corresponding input images as training data sets corresponding to the step they illustrate by the training dataset identifier. If the resulting similarity score is less than the predetermined threshold, the vector maps and their corresponding input images are discarded.”; the model gives feedback to the user visually with instructions to the user if the placement of an object in a product assembly process is incorrect when the worker is assembling the product; the continuously updated visual indication to the worker is part of the trained inspection model; therefore the re-training after each iteration is based on the current image in worker’s visual field, the feedback shown to the user, and the classification codes or “template” that allow model to know visually what specific step of the product assembly process the worker is at).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the processor, as taught by Singh, to be programmed to associate each of the plurality of training images with one of the stores classification codes, output the classification codes by the trained inspection model based on the input of the current image, and further train, via machine learning, the trained inspection model based on the current image and the classification code output by the trained inspection model based on the input of the current image, as taught by Guo.
The suggestion/motivation for doing so would have been that “while automation has become ubiquitous in manufacturing, there remain various functions or steps that rely on human intervention; for example, an assembly line worker may, during a step in the assembly of an appliance door, misalign one or more components of the door assembly, which may not be readily discernable by human inspection; such misalignment may be problematic to downstream steps in the product assembly process or may manifest as a failure in production (e.g., resulting in warranty claims and/or customer dissatisfaction); as such, manufacturers continue to seek greater efficiency in manufacturing outcomes by minimizing product assembly deviation (e.g., reducing or eliminating human error introduced by the human component of the process).” (Guo, para. [0019]).
Therefore, it would have been obvious to combine Singh, with Guo, to obtain the invention as specified in claim 1.
Regarding claim 2, Singh, in view of Guo, teaches the wearable inspection unit of Claim 1, wherein the media output component is configured to display an instruction for the current step to the user via the augmented reality overlay (Singh, para. [0038]; FIG. 3; see rejection of claim 1 above showing the display of instructions/indications/guidance of what current step in the manufacturing process the user of the augmented reality device is at; para. [0017]: ““The process map may be used to generate intuitive and efficient instructions for both novice and expert operators to aid and navigate the operator through the process. These instructions may also be delivered through the same AR device that captures the data. Thus, the AR device serves both as the data capture device for input to the learning system and as the content delivery device for the instructions generated by the learning system.”).
Regarding claim 3, Singh, in view of Guo, teaches the wearable inspection unit of Claim 1, wherein the at least one processor is further programmed to display feedback associated with the current step via the augmented reality overlay (Singh, para. [0038]; FIG. 3; see rejection of claim 1 above; the indicator “T0907” is feedback displayed for the user on the augmented reality display for the current step of the process).
Regarding claim 4, Singh, in view of Guo, teaches the wearable inspection unit of Claim 1, wherein the at least one sensor configured to capture images or video based on the current view of the user (Singh, para. [0041], lines 8-12; para. [0043], lines 1-3: “Although not shown in FIG. 4, the method may further include performing object recognition on the received image data to identify and track objects in the user's field of view, and generating encoded data of the manual industrial operation being performed representing one or more state changes of the manual industrial operation based on the object recognition.”; “In some embodiments, the output AR components output in 440 may indicate a suggested path for performing the manual industrial operation within a field of view of the user.”).
Regarding claim 5, Singh, in view of Guo, teaches the wearable inspection unit of Claim 1, wherein the at least one processor is further programmed to: receive a first image from the at least one sensor Singh, para. [0039]; FIG. 4; see rejection of claim 1 above);
determine a first step associated with the first image (Singh, para. [0040]; FIG. 4; para. [0038]; FIG. 3 see rejection of claim 1 above);
subsequently receive a second image from the at least one sensor (Singh, para. [0024], lines 12-18: “The stream of data may include images, audio, video, spatial data, temperature, and the like, captured by the AR device 120 in real-time and provided to the AR server 110. The images captured by the AR device 120 may include pictures or video of the user performing the process with respect to the machine or equipment.”; Singh receives a continuous stream of images or video frames taken from where the user is observing with the AR glasses/headset); and
determine a second subsequent step associated with the second image (Singh, para. [0041]; FIG. 4: “In 430, the method further includes determining a future state of the manual industrial operation that will be performed by the user based on the current state, and generating one or more augmented reality (AR) components based on the future state of the manual industrial operation. Here, the future state of the manual industrial operation may be performed by a learning system of the AR software. Although not shown in FIG. 4, the method may further include performing object recognition on the received image data to identify and track objects in the user's field of view, and generating encoded data of the manual industrial operation being performed representing one or more state changes of the manual industrial operation based on the object recognition. Furthermore, the encoded data may be input to the learning system that continuously receives and learns from the encoded data of the operation being performed, predicts state changes that will occur for the operation based on the learning, and determines the future state of the operation based on the predicted state changes.”;
PNG
media_image12.png
414
418
media_image12.png
Greyscale
).
Regarding claim 7, Singh, in view of Guo, teaches the wearable inspection unit of Claim 1, wherein the process is installation of a part, and wherein the at least one processor is further programmed to: determine if the part was properly installed based on the current image; and provide feedback based on whether or not the part was properly installed via the augmented reality overlay (Singh, para. [0023]; para. [0038]: lines 25-27: “For example, the augmented reality components can indicate a process path or a next part in the operation that is to be replaced/inspected.; “In the scene construction module 240, the placement of parts in an electrical cabinet assembly is evaluated against a part layout using holographic indicators. Simple holograms may be provided to indicate when a part is present, but not detected, detected but not properly placed, or detected and properly placed. These holograms and their placement may be packaged for and rendered on the AR device 210 (e.g., HoloLens) in real-time.”).
Regarding claim 8, Singh, in view of Guo, teaches the wearable inspection unit of Claim 1, further comprising an attachment system for attaching the wearable inspection unit to the user (Singh, para. [0039], lines 24-28; para. [0045], lines 2-4; para. [0022], lines 11-13: “The image data may be captured by an AR device such as a pair of glasses, a helmet, a band, a camera, and the like, which may be worn by or attached to the user.; “According to various embodiments, the storage 540 may store image data captured of a manual industrial operation being performed by a user. Here, the image data may be captured by an AR device being worn by the user, attached to the user, or associated with the manual industrial operation.”; “The AR device 120 may be one or more of glasses, a helmet, a screen, a camera, a microphone, and/or the like, which are associated with the user 10. In some examples, the AR device 120 or a plurality of AR devices may be attached to or worn by the user 10.”; FIG. 1; see rejection of claim 1 above to see glasses with attachment portion to the user’s face).
Regarding claim 9, Singh teaches a system comprising: (Singh, para. [0022]: “FIG. 1 illustrates an augmented reality system 100 in accordance with an example embodiment. In this example, a user 10 performs operations on one or more types of industrial assets 130 which may include machine and equipment in the fields of transportation, energy, healthcare, manufacturing, and the like. Referring to FIG. 1, the system 100 includes an augmented reality (AR) server 110 in communication with an AR device 120 associated with the user 10. The AR server 110 may be a cloud platform, a server, or another computing device attached to a network. The AR device 120 may be one or more of glasses, a helmet, a screen, a camera, a microphone, and/or the like, which are associated with the user 10. In some examples, the AR device 120 or a plurality of AR devices may be attached to or worn by the user 10. As another example, the AR device 120 may be within a field of view of the user 10 but not attached to the user. The AR server 110 and the AR device 120 may be connected to each other by a network such as the Internet, private network, or the like. As another example, the AR device 120 may be connected to the AR server 110 by a cable or the AR device 120 may incorporate the features of the AR server 110 within the AR device 120.”;
PNG
media_image3.png
515
736
media_image3.png
Greyscale
)
a wearable inspection unit comprising at least one sensor configured to capture images based on a current view of a wearer (Singh, para. [0023]; see wearable AR glasses above in FIG. 1 above: “The AR device 120 may be outfitted with one or more data gathering components (e.g., cameras, sensors, LIDAR, thermal cameras, etc.) which are capable of capturing images, spatial data, audio, temperature, and the like, and which are configured to monitor respective operations or conditions of the user 10 performing operations with respect to an asset 130. Data captured by the AR device 120 can be recorded and/or transmitted to the AR server 120 or other remote computing environment described herein. By bringing the data into the AR system 100, the AR platform described herein which may include software or a combination of hardware and software may analyze a process being performed by the user 10 with respect to the asset 130 and provide augmented reality components that are related to the process. The AR software may be included in the AR server 110, the AR device 120, or a combination thereof. As a non-limiting example, the user 10 may be performing a maintenance process, a repair process, a cleaning process, a production/assembly process, or any other process known in which a user interacts with machines or equipment in an industrial setting. The AR server 120 may analyze the captured data and determine a current state of the process being performed by the user. Furthermore, the AR server 110 can provide augmented reality components to the AR device 120 based on a future state of the process being performed by the user 10. For example, the augmented reality components can indicate a process path or a next part in the operation that is to be replaced/inspected.”);
a media output component configured to display an augmented reality overlay to the wearer (Singh, para. [0027]: “FIG. 2 illustrates an augmented reality process 200 in accordance with an example embodiment. In this example, the augmented reality process 200 includes a plurality of components including an AR device 210 that captures process data and provides the process data to an object recognition module 220. The object recognition module performs object recognition from the data and provides the object recognized data to a process learning module 230. The process learning module determines a state of a manual industrial process 250 (or operation) and provides data about the state to a scene construction module 240. The scene construction module 240 generates AR components for display by the AR device 210 based on a scene in which a user/operator is performing the process 250. Here, the scene construction module may overlay holographic components within a field of view of the user/operator wearing the AR device 210 and feedback the AR components to the AR device 210. FIG. 2 also illustrates that the manual industrial process 250 performed by the user/operator includes a plurality of steps.”;
PNG
media_image4.png
405
796
media_image4.png
Greyscale
); and
a controller in communication with the wearable, wherein the controller comprises at least one processor in communication with at least one memory device, wherein the at least one processor programmed to: (Singh, para. [0044]: “FIG. 5 illustrates a computing system 500 for generating an augmented reality in accordance with an example embodiment. For example, the computing system 500 may be a cloud platform, a server, a user device, or some other computing device with a processor. Also, the computing system 500 may perform the method of FIG. 4. Referring to FIG. 5, the computing system 500 includes a network interface 510, a processor 520, an output 530, and a storage device 540. Although not shown in FIG. 5, the computing system 500 may include other components such as a display, an input unit, a receiver/transmitter, and the like. The network interface 510 may transmit and receive data over a network such as the Internet, a private network, a public network, and the like. The network interface 510 may be a wireless interface, a wired interface, or a combination thereof. The processor 520 may include one or more processing devices each including one or more processing cores. In some examples, the processor 520 is a multicore processor or a plurality of multicore processors. Also, the processor 520 may be fixed or it may be reconfigurable. The output 530 may output data to an embedded display of the device 500, an externally connected display, an AR device, a cloud instance, another device or software, and the like. The storage device 540 is not limited to any particular storage device and may include any known memory device such as RAM, ROM, hard disk, and the like.”;
PNG
media_image5.png
395
422
media_image5.png
Greyscale
).
Regarding the remaining limitations of claim 9, as well as dependent claims 10, 12, and 14 they recite the functions of the apparatuses of claims 1-2, 5, and 7 respectively as different apparatuses (independent claim 1 recites a wearable inspection unit while independent claim 9 recites a system including a wearable inspection unit). Thus, the analyses in rejecting claims 1-2 and 5-7 are equally applicable to the remaining limitations of claim 9 and dependent claims 10-14, respectively.
Regarding claim 15, Singh, in view of Guo, teaches the system of Claim 9, wherein the controller is in communication with a visual classifier server, and wherein the at least one processor is further programmed to: transmit the current image to the visual classifier server; and receive the classification code from the visual classifier server (Singh, FIG. 1; para. [0025]-[0026]: “According to various embodiments, the AR server 110 can analyze the images and/or audio coming in and determine a current state of the process being performed by the user 10 based on the analyzed images/audio with respect to a one or more models maintained by the AR server 110. For example, the AR server 110 may maintain a process map including images of the process performed previously by the user 10 or other users as well as descriptions, images, and audio of the individual steps/phases of the process being performed by the user 10. The AR server 110 may determine augmented reality components to output based on a state of the process. For example, the AR server 110 may determine augmented reality components to output based on a previous state, a current state and/or a future state of