DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Responsive to the communication dated 02/11/2026
Claims 1-2 and 4-8 are presented for examination
Finality
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Response to Arguments- Claim Objections
Applicant’s arguments, see page 6, filed 02/11/2026, with respect to the objection to claims 1-8 have been fully considered and are persuasive. The objection to claims 1-8 has been withdrawn.
Response to Arguments- 35 USC § 112
Applicant’s arguments, see page 6, filed 02/11/2026, with respect to the rejection of claims 1-8 under 112(b) have been fully considered and are persuasive. The rejection of claims 1-8 under 112(b) has been withdrawn.
However, it should be noted that the newly amended claim language introduces new issues under 112 for claim 1, and claims 1-8 are now rejected under 112(a).
Response to Arguments- 35 USC § 101
Applicant’s arguments, see page 7, filed 02/11/2026, with respect to the rejection of claims 1-8 under 101. have been fully considered and are persuasive. The rejection of claims 1-8 under 101 has been withdrawn.
Particularly, in view of the new amendments and the recent training and guidance, the claims are integrated into a practical application as they not only identify the non-conformances of the physical aircraft component, but positively recite steps for actively facilitating the repair of these non-conformances using information generated by the claimed process.
Response to Arguments- 35 USC § 103
Applicant's arguments filed 02/11/2026 have been fully considered but they are not persuasive.
Applicant argues that no prior art teaches determining coordinates of at least three virtual calibration points in the first reference frame which are positioned on the virtual component inside of an outer parameter of the virtual component
Examiner responds by firstly noting that as objected to later in this action, the term “outer parameter” is interpreted as being a misspelling of -“outer perimeter” and is treated as such for the purposes of this examination.
Examiner further responds by explaining that these features are taught by the previously cited prior art in combination with newly cited reference Augmented Reality-based factory planning- an application tailored to industrial needs (Hereinafter Pentenrieder)
Particularly, Scott makes obvious ([Par 63-65] “In one illustrative example, in localizing portable computing device 214 with model 244, localization application 222 locates location target 250 for physical object 204. Localization application 222, uses the target position 252 of location target 250 to localize portable computing device 214 with model 244 of the region 240 of physical object 204 in defined coordinate cube 246 and identifies position 248 of portable computing device 214 with respect to physical object 204. In this illustrative example, location target 250 is a physical feature. Location target 250 can be, for example, a plate, a barcode, a radio frequency identifier (RFID) device, an anchor, a spatial anchor, anchor plates, or some other suitable structure. Location target 250 is in a known location on physical object 204 and has a corresponding location in model 244 representing physical object 204. The location is in three dimensions and can be target position 252 if an orientation of location target 250 is also known. The position of location target 250 can be correlated with the corresponding location in model 244 to identify the position 248 of portable computing device 214 with respect to physical object 204.”)
Coding Labs makes obvious wherein the step of determining the transfer matrix comprises ([Page 1 Par 1-2] “In this article we will try to understand in details one of the core mechanics of any 3D engine, the chain of matrix transformations that allows to represent a 3D object on a 2D monitor. We will try to enter into the details of how the matrices are constructed and why, so this article is not meant for absolute beginners. I will assume general knowledge of vectors math and matrices math. We will first talk about the relationship between transformations and vector spaces. Then we will show how a transformation can be represented in matrix form. From there we will show the typical sequence of transformations that you will need to apply, which is from Model to World Space, then to Camera and then Projection.” [Page 5 Par 1 – Page 6 Par 2] “With all the objects at the right place we now need to project them to the screen. This is usually done in two steps. The first step moves all the object in another space called the View Space. The second step performs the actual projection using the projection matrix … How do we calculate the transformation matrix for View Space? Now, if you imagine you want to put the camera in World Space you would use a transformation matrix that is located where the camera is and is oriented so that the Z axis is looking to the camera target. The inverse of this transformation, if applied to all the objects in World Space, would move the entire world into View Space. Notice that we can combine the two transformations Model To World and World to View into a single transformation Model To View…. The scene is now in the most friendly space possible for a projection, the View Space. All we have to do now is to project it onto the imaginary screen of the camera. Before flattening the image, we still have to move into another, final space, the Projection Space. This space is a cuboid which dimensions are between -1 and 1 for every axis. This space is very handy for clipping (anything outside the 1:-1 range is outside the camera view area) and simplifies the flattening operation (we just need to drop the z value to get a flat image). To go from the View Space into the Projection Space we need another matrix, the View to Projection matrix, and the values of this matrix depend on what type of projection we want to perform. The two most used projections are the Orthographic Projection and the Perspective Projection. To do the Orthographic projection we have to define the size of the area that the camera can see. This is usually defined with a width and height values for the x and y axis, and a near and far z values for the z axis … Given these values we can create the transformation matrix that remaps the box area into the cuboid. The matrix that follows is transforms vectors from View Space into Ortho Projected Space and assumes a right handed coordinates system.”)
The combination of Scott, Schmirler, and Coding Labs does not explicitly teach determining coordinates of at least three calibration points which are positioned on the component inside of an outer parameter of the component;
Pentenrieder makes obvious determining coordinates of at least three calibration points which are positioned on the component inside of an outer parameter of the component; ([Figure 5] Shows an overview of translating physical calibration marker positions into model-space positions [Figure 10] Shows an example calibration marker setup with [Figure 11] Showing the scene overlaid with aligned model geometry. As can be seen, there are more than 3 markers and they are not placed only at the perimeter of the space. [Page 3 Col 2 Par 5 – Page 4 Col 1 Par 1] “Roivis was created as a marker based AR software with a graphical user interface tailored to the AR-based planning process. The usage of a marker based tracking offers an easy interface where only a printed marker and a camera are needed to start planning with the software. The crucial aspects related to the acquisition of the planning input data are the accurate positioning of the marker (which will be discussed in more detail in section 5) and the generation of applicable image data. The image data must allow successful tracking (sufficient lighting conditions and fully visible markers) and has to show the important views of the scene regarding the planning results. Given the necessary input data, planning with Roivis can be realized in a few steps. The image data is loaded by choosing the picture source. 3D models in VRML 2.0 format can be added to the scene and bound directly to a marker or to a coordinate system defined relatively to a marker using a so called offset configuration. Files for camera calibration and marker configuration data are loaded to complete the scene. The former contains the camera resolution and information on the intrinsic camera parameters. Whereas the latter defines the markers, which are used for the planning process: their size, identification number and possible offsets to reference the marker to an external coordinate system (e.g. a part coordinate system). Finally, manipulation tools are available to move and rotate the 3D models if needed.”) [Figure 6] Shows an example of wireframe virtual geometry matched to corresponding physical geometry using at least 3 calibration markers/points)
PNG
media_image1.png
577
434
media_image1.png
Greyscale
PNG
media_image2.png
397
535
media_image2.png
Greyscale
PNG
media_image3.png
451
527
media_image3.png
Greyscale
PNG
media_image4.png
396
582
media_image4.png
Greyscale
Pentenrieder is analogous art because it is within the field of industrial augmented reality and physical-virtual component matching and tracking. It would have been obvious to one of ordinary skill in the art to combine Pentenrieder with Scott, Schmirler, and Coding Labs before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to better manage repair and manufacturing applications, particularly by enabling more accurate tracking. Pentenrieder notes how tracking error is the one of the biggest causes of frustration in the field and one of the most significant factors hindering widespread adoption of such industrial augmented reality technology. ([Page 2 Col 1 Par 3-5] “Input and influence factors Most of the demonstrator applications and prototypes of AR-based industrial applications did not achieve the goal of becoming a well-accepted and often used solution in the everyday product life cycle. Possible reasons for this lack of approval can be found in [22]. Here, Regenbrecht et al. state that the maturity of contributing technologies (tracking, dis plays, content generation, wearable computing, etc.) does not suit the demanding industrial conditions yet regarding robustness, reliability, quality and practical experience. Other than the factors mentioned above, influencing elements like usability and ergonomics need to be considered for a successful industrial AR application. There are many studies and analysis on the quality of different components of AR-systems such as tracking analysis [27], [19] or display evaluations [1], [15]. One large concern is the accuracy of the overall system representing the preciseness of the virtual overlay onto the users field of view. For statements on the overall system accuracy, several influence factors need to be considered. Holloway lists tracking, calibration and modeling as main error sources in Augmented Reality systems [12]. Approaches for the combination of different sources of error and error propagation can also be found. For instance in Coelho et al. [4], a scene graph with uncertain transforms is presented, whose sources of error are also based on the work of Holloway.”) To this end, Pentenrieder presents a method for more accurate tracking between physical objects and 3D models in industrial contexts. ([Page 3 Col 2 Par 4] “Roivis was created as a marker based AR software with a graphical user interface tailored to the AR-based planning process. The usage of a marker based tracking offers an easy interface where only a printed marker and a camera are needed to start planning with the software. The crucial aspects related to the acquisition of the planning input data are the accurate positioning of the marker (which will be discussed in more detail in section 5) and the generation of applicable image data. The image data must allow successful tracking (sufficient lighting conditions and fully visible markers) and has to show the important views of the scene regarding the planning results.”) Overall, one of ordinary skill in the art would have recognized that combining Pentenrieder with Scott, Schmirler, and Coding Labs would result in a system that is significantly more accurate.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-2 and 4-8 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites “at least three virtual calibration points in the first reference frame which are positioned on the virtual component inside of an outer parameter of the virtual component” (Note that “parameter” is interpreted as being a misspelling of the word “perimeter”) There does not seem to be any support in the disclosure for the calibration points being positioned “inside of an outer parameter of the virtual component.” The only positions for these points described in the disclosure merely generally describe them as being “on the virtual component” ([Par 53-54] “This phase of determining the transfer matrix comprises a step of selecting at least three virtual calibration points P1, P2, P3 located in the first reference frame R1 on the virtual model 10 of the aircraft, and a step of determining the coordinates of each virtual calibration point P 1, P2, P3 in the first reference frame R1. As shown in Figure 2, the virtual calibration points P1, P2, P3 are positioned on the virtual component 12. For each virtual calibration point P1, P2, P2, the real component 28 has a real calibration point PR1, PR2, PR3 which corresponds to said virtual calibration point P1, P2, P3. The virtual calibration points P1, P2, P2 are noteworthy points, chosen so that the corresponding real calibration points PR1, PR2, PR3 are easily identifiable on the real component 28.”)
While the arguments cite to Fig. 2 as providing support, the points shown in the figure are on the “perimeter” of the depicted geometry. Because 3D space must be considered, the entire surface of a volume is effectively its “perimeter.” With this in mind, it is clear that these points are on the surface of the volume and therefore on its perimeter. Because of this, this figure is insufficient to show support for the amended language of “at least three virtual calibration points in the first reference frame which are positioned on the virtual component inside of an outer parameter of the virtual component.”
PNG
media_image5.png
619
778
media_image5.png
Greyscale
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(1) Claims 1 and 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over Scott (US 20200118345 A1) in view of Schmirler (US 20180131907 A1) in further view of Coding Labs - Article - World, View and Projection Transformation Matrices (Hereinafter Coding Labs) as well as Augmented Reality-based factory planning- an application tailored to industrial needs (Hereinafter Pentenrieder)
Claim 1. Scott teaches A method of repairing a real component of an aircraft having at least one non-conformance, ([Par 8] “The augmented reality application, when executed, plots points in the defined coordinate cube corresponding to nonconformance locations on the physical object; determines a sub-set of the points plotted in the defined coordinate cube that are visible in an image of the region of the physical object acquired by the portable computing device where the sub-set excludes plotted points determined to be located beyond the physical object structure when viewed at the position of the portable computing device; and displays the nonconformance data for the sub-set of the points visible in the image in association with a sub-set of the nonconformance locations for the physical object in the image displayed on the display system in the portable computing device.” [Par 2] “The nonconformance can be, for example, a crack, a scratch, a delamination, an incorrect dimension, an incorrect color, or some other feature that is incorrect.” [Par 95] “In this illustrative example, selecting a graphical indicator can result in the display of additional information about nonconformances. For example, selection of the starburst 434 for nonperformance location 414 results in the display of nonconformance data 440 in window 442. Nonconformance data 440 can take a number of different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance”)said real component corresponding to a virtual component defined in a first reference frame and having at least one criterion area, each criterion area being defined in the first reference frame and ([Par 61] “In this example, modeling application 220 can generate model 244 of the region 240 of physical object 204 in defined coordinate cube 246 using point cloud representation 242 for region 240 of physical object 204.” [Examiner’s note: the region of the object is interpreted as the criterion area])having at least one acceptance criterion, ([Par 95] “In this illustrative example, selecting a graphical indicator can result in the display of additional information about nonconformances. For example, selection of the starburst 434 for nonperformance location 414 results in the display of nonconformance data 440 in window 442. Nonconformance data 440 can take a number of different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance” [Examiner’s note: the instructions to resolve the nonconformance is interpreted as being equivalent to the acceptance criterion]) characterized in that the method comprises: displaying the real component with an augmented reality device, ([Par 34] “Nonconformance data can be displayed on the live view to provide an augmented reality display. The nonconformance data can be displayed by overlay over the live view of fuselage section 102 using an augmented reality application running on tablet computer 108. A live view is a view of interior 106 of fuselage section 102 while tablet computer 108 is in or near interior 106 of fuselage section 102. In other words, images of interior 106 of fuselage section 102 are generated and augmented with overlaid nonconformance data. As tablet computer 108 changes position, the image changes to show the view from the current position of tablet computer 108. The images for live view can be displayed as a video in real time. In other words, human operator 104 can see what is currently in the view the camera of tablet computer 108.”) configured to display a pointer superimposed on the real component and for determining coordinates of the pointer ([Par 92] “The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances. In this illustrative example, nonconformances that are visible in image 400 are augmented with graphical indicators while nonconformances that are obscured in image 400 are not identified using graphical indicators.” [Par 94] “In this illustrative example, the visible nonconformance locations are identified in image 400 by augmenting the live view of an image 400 with graphical indicators to identify the nonconformance locations. In this example, the graphical indicators are starbursts. As depicted, nonconformance location 410 is indicated using starburst 430; nonconformance location 412 is indicated using starburst 432; nonconformance location 414 is indicated using starburst 434; and nonconformance location 416 is indicated using starburst 436”) ([Par 66] “In this example, augmented reality application 224 plots points 234 in defined coordinate cube 246 corresponding to nonconformance locations 208 on physical object 204. Augmented reality application 224 determines sub-set 254 of points 234 plotted in defined coordinate cube 246 that are visible in image 238 of region 240 of physical object 204 acquired by portable computing device 214 at position 248 of portable computing device 214.”) ([Par 61] “In this example, modeling application 220 can generate model 244 of the region 240 of physical object 204 in defined coordinate cube 246 using point cloud representation 242 for region 240 of physical object 204.”) positioning the pointer to target the non-conformance, ([Par 92] “The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances. In this illustrative example, nonconformances that are visible in image 400 are augmented with graphical indicators while nonconformances that are obscured in image 400 are not identified using graphical indicators.” [Par 94] “In this illustrative example, the visible nonconformance locations are identified in image 400 by augmenting the live view of an image 400 with graphical indicators to identify the nonconformance locations. In this example, the graphical indicators are starbursts. As depicted, nonconformance location 410 is indicated using starburst 430; nonconformance location 412 is indicated using starburst 432; nonconformance location 414 is indicated using starburst 434; and nonconformance location 416 is indicated using starburst 436”) determining coordinates in the first reference frame of a ([Par 92] “The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances.” [Par 94] “In this illustrative example, the visible nonconformance locations are identified in image 400 by augmenting the live view of an image 400 with graphical indicators to identify the nonconformance locations. In this example, the graphical indicators are starbursts. As depicted, nonconformance location 410 is indicated using starburst 430”) ([Par 92] “The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances.”)determining whether the ([Par 91-95] “In this illustrative example, nonconformance locations are present in flight deck 402. In this example, these nonconformance locations include nonconformance location 410, nonconformance location 412, nonconformance location 414, nonconformance location 416, nonconformance location 418, nonconformance location 420, and nonconformance location 422. These nonconformance locations can be identified from nonconformance data and model 300 which includes details for flight deck 402. The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances. In this illustrative example, nonconformances that are visible in image 400 are augmented with graphical indicators while nonconformances that are obscured in image 400 are not identified using graphical indicators. As depicted, nonconformance location 410, nonconformance location 412, nonconformance location 414, and nonconformance location 416 are visible in image 400. However, nonconformance location 418, nonconformance location 420, and nonconformance location 422 are obscured. In this depicted example these nonconformance locations are obscured by chair 424 in this view of flight deck 402 in image 400. In this illustrative example, the visible nonconformance locations are identified in image 400 by augmenting the live view of an image 400 with graphical indicators to identify the nonconformance locations. In this example, the graphical indicators are starbursts. As depicted, nonconformance location 410 is indicated using starburst 430; nonconformance location 412 is indicated using starburst 432; nonconformance location 414 is indicated using starburst 434; and nonconformance location 416 is indicated using starburst 436. Graphical indicators are not displayed on image 400 for nonconformance location 418, nonconformance location 420, and nonconformance location 422 because these locations are obscured from this view of flight deck 402. In this illustrative example, selecting a graphical indicator can result in the display of additional information about nonconformances. For example, selection of the starburst 434 for nonperformance location 414 results in the display of nonconformance data 440 in window 442. Nonconformance data 440 can take a number of different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance”) and repairing the non-conformance according to the acceptance criterion, in order to make the real component of the aircraft acceptable, ([Par 2] “The nonconformance can be, for example, a crack, a scratch, a delamination, an incorrect dimension, an incorrect color, or some other feature that is incorrect.” [Par 95] “In this illustrative example, selecting a graphical indicator can result in the display of additional information about nonconformances. For example, selection of the starburst 434 for nonperformance location 414 results in the display of nonconformance data 440 in window 442. Nonconformance data 440 can take a number of different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance”) ([Par 63-65] “In one illustrative example, in localizing portable computing device 214 with model 244, localization application 222 locates location target 250 for physical object 204. Localization application 222, uses the target position 252 of location target 250 to localize portable computing device 214 with model 244 of the region 240 of physical object 204 in defined coordinate cube 246 and identifies position 248 of portable computing device 214 with respect to physical object 204. In this illustrative example, location target 250 is a physical feature. Location target 250 can be, for example, a plate, a barcode, a radio frequency identifier (RFID) device, an anchor, a spatial anchor, anchor plates, or some other suitable structure. Location target 250 is in a known location on physical object 204 and has a corresponding location in model 244 representing physical object 204. The location is in three dimensions and can be target position 252 if an orientation of location target 250 is also known. The position of location target 250 can be correlated with the corresponding location in model 244 to identify the position 248 of portable computing device 214 with respect to physical object 204.”)
Scott does not explicitly teach coordinates in a second reference frame; determining a transfer matrix for converting coordinates of a point located in the second reference frame into coordinates of a virtual point in the first reference frame; determining coordinates in the first reference frame of a virtual point based on the transfer matrix and the coordinates in the second reference frame of the associated point; a virtual point belonging to a criterion area; displaying the acceptance criteria associated with the criterion area of the virtual point to address the non-conformance according to the acceptance criterion; wherein the step of determining the transfer matrix comprises determining coordinates of at least three calibration points which are positioned on the component inside of an outer parameter of the component; points in the second reference frame; determining coordinates in the second reference frame for points; determining the transfer matrix based on points in the second reference frame;
Schmirler makes obvious ([Par 141] “Once at the location, further directional arrows can be generated that indicate the particular industrial device, machine, or machine component experiencing the issue. Again, the direction and display locations for these arrows are based on the current location and orientation data 606”) displaying the acceptance criteria associated with the criterion area of the virtual point to address the non-conformance according to the acceptance criterion; ([Par 144] “As noted above, some embodiments of VR/AR presentation system 302 can be configured to assist users in connection with addressing detected operational or maintenance issue using interactive workflow presentations customized for the detected issue.” [Par 152-155] “rendering component 308 send workflow presentation data 1602 to each recipient's wearable appliance 206. In an example scenario, rendering component 308 can render the workflow presentation as an augmented reality presentation that renders a sequence of instructions as an overlay on the user's field of view. These presentations can include graphical indicator overlays that point to or otherwise visually identify devices, workstations, or machine components that the user's attention should be focused on during a current step of the workflow, alphanumeric instructions that inform the user of the next step to be performed, feedback graphics that indicate when the step has been correctly performed or when the user has deviated from the proper workflow sequence, and other such information … Concurrently, device interface component 314 collects user data 1704 that can be used to confirm that the user is performing the steps recommended by the workflow delivered to the user's wearable appliance 206. … Other user data 1704 may include device orientation data identifying a current orientation of the user's wearable appliance 206, which, when combined with the user's current location data, can indicate whether the user is currently viewing a correct area of the automation system for completion of a pending workflow step.”)
Schmirler is analogous art because it is within the field of augmented reality within the context of manufacturing. It would have been obvious to one of ordinary skill in the art to combine Schmirler with Scott before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to enable active repair of nonconformances. As noted by Schmirler, identifying and properly determining how to address issues in a manufacturing context often takes a great deal of time and effort ([Par 49] “When diagnosing problems, maintenance personnel are often required to search several of these sources of information individually, using several different software packages specific to the respective data sources being searched. Moreover, searching for information pertaining to a particular device or machine often requires an extensive knowledge of the overall industrial system in order to locate the data source to be searched (e.g., in order to locate the appropriate industrial controller or HMI terminal), as well as to identify the relevant operator screens and control program routines. Individually searching each of these data sources in connection with solving a system downtime issue or other problem can delay correction of maintenance issues, resulting in lost revenue and scheduling problems. Also, if an operator or maintenance person is not near an information source—such as an HMI terminal—at the time an operational or maintenance issue occurs, the user may not be notified of the issue in a timely fashion.”) To this end, Schmirler presents a system that enables the easy identification and correction of manufacturing issues using augmented reality ([Par 50] “To address these and other issues, one or more embodiments of the present disclosure provide a system that generates and delivers augmented reality (AR) or virtual reality (VR) presentations [Par 52] “… the VR/AR presentation system can provide automation system data, notifications, and proactive guidance to the user via modification of the user's view of his or her immediate surroundings. Such modifications can include, for example, superimposing data values or indicators on a user's view of a machine or automation system through the user's wearable computer (or other client device capable of rendering a substantially real-time view of the machine or system). The system can customize presentation of this information based on the user's role, location, line of sight, type of wearable device, and/or other contextual information.” [Par 144] “As noted above, some embodiments of VR/AR presentation system 302 can be configured to assist users in connection with addressing detected operational or maintenance issue using interactive workflow presentations customized for the detected issue.” [Par 152-155] “rendering component 308 send workflow presentation data 1602 to each recipient's wearable appliance 206. In an example scenario, rendering component 308 can render the workflow presentation as an augmented reality presentation that renders a sequence of instructions as an overlay on the user's field of view. These presentations can include graphical indicator overlays that point to or otherwise visually identify devices, workstations, or machine components that the user's attention should be focused on during a current step of the workflow, alphanumeric instructions that inform the user of the next step to be performed, feedback graphics that indicate when the step has been correctly performed or when the user has deviated from the proper workflow sequence, and other such information”) Overall, one of ordinary skill in the art would have recognized that combining Schmirler with Scott would result in a system that not only enables the detection of nonconformances, but also the easy correction of them.
The combination of Scott and Schmirler does not explicitly teach coordinates in a second reference frame; determining a transfer matrix for converting coordinates of a point located in the second reference frame into coordinates of a virtual point in the first reference frame; determining coordinates in the first reference frame of a virtual point based on the transfer matrix and the coordinates in the second reference frame of the associated point; wherein the step of determining the transfer matrix comprises determining coordinates of at least three calibration points which are positioned on the component inside of an outer parameter of the component; points in the second reference frame; determining coordinates in the second reference frame for points; determining the transfer matrix based on points in the second reference frame;
Coding Labs makes obvious coordinates in a second reference frame; determining a transfer matrix for converting coordinates of a point located in the second reference frame into coordinates of a virtual point in the first reference frame; determining coordinates in the first reference frame of a virtual point based on the transfer matrix and the coordinates in the second reference frame of the associated point; wherein the step of determining the transfer matrix comprises ([Page 1 Par 1-2] “In this article we will try to understand in details one of the core mechanics of any 3D engine, the chain of matrix transformations that allows to represent a 3D object on a 2D monitor. We will try to enter into the details of how the matrices are constructed and why, so this article is not meant for absolute beginners. I will assume general knowledge of vectors math and matrices math. We will first talk about the relationship between transformations and vector spaces. Then we will show how a transformation can be represented in matrix form. From there we will show the typical sequence of transformations that you will need to apply, which is from Model to World Space, then to Camera and then Projection.” [Page 5 Par 1 – Page 6 Par 2] “With all the objects at the right place we now need to project them to the screen. This is usually done in two steps. The first step moves all the object in another space called the View Space. The second step performs the actual projection using the projection matrix … How do we calculate the transformation matrix for View Space? Now, if you imagine you want to put the camera in World Space you would use a transformation matrix that is located where the camera is and is oriented so that the Z axis is looking to the camera target. The inverse of this transformation, if applied to all the objects in World Space, would move the entire world into View Space. Notice that we can combine the two transformations Model To World and World to View into a single transformation Model To View…. The scene is now in the most friendly space possible for a projection, the View Space. All we have to do now is to project it onto the imaginary screen of the camera. Before flattening the image, we still have to move into another, final space, the Projection Space. This space is a cuboid which dimensions are between -1 and 1 for every axis. This space is very handy for clipping (anything outside the 1:-1 range is outside the camera view area) and simplifies the flattening operation (we just need to drop the z value to get a flat image). To go from the View Space into the Projection Space we need another matrix, the View to Projection matrix, and the values of this matrix depend on what type of projection we want to perform. The two most used projections are the Orthographic Projection and the Perspective Projection. To do the Orthographic projection we have to define the size of the area that the camera can see. This is usually defined with a width and height values for the x and y axis, and a near and far z values for the z axis … Given these values we can create the transformation matrix that remaps the box area into the cuboid. The matrix that follows is transforms vectors from View Space into Ortho Projected Space and assumes a right handed coordinates system.”)
Coding Labs is analogous art because it is within the field of 3D rendering. It would have been obvious to one of ordinary skill in the art to combine it with Scott and Schmirler before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to effectively display the 3D imagery disclosed by Scott and Schmirler on a display. As explained by Coding Labs, the ability to perform the projection of 3D objects into 2D coordinates for display on a screen is one of the fundamental mechanics of 3D rendering ([Page 1 Par 1] “In this article we will try to understand in details one of the core mechanics of any 3D engine, the chain of matrix transformations that allows to represent a 3D object on a 2D monitor. We will try to enter into the details of how the matrices are constructed and why, so this article is not meant for absolute beginners. I will assume general knowledge of vectors math and matrices math.”) Further, Coding Labs discloses the benefits afforded by using efficient modern rendering engines ([Page 6 Par 3 – Page 7 Par 1] “To complete the transformation we will need to divide every component of the vector by the w component itself. Current graphics APIs do the division for you, therefore you can simply multiply all your vertices by the perspective projection matrix and send the result to the GPU. GPU takes care of dividing by w, clipping those vertices outside the cuboid area, flattening the image dropping the z component, re-mapping everything from the -1 to 1 range into the 0 to 1 range and then scale it to the viewport width and height, and rasterizing the triangles to the screen (if you are doing the rasterization on the CPU you will have to take care of these steps yourself). We can therefore take these last steps for granted if we render via OpenGL or DirectX, so the perspective space is the last step of our chain of transformation.”) Overall, one of ordinary skill in the art would have recognized that combining. Coding Labs with Scott and Schmirler would result in a system that allowed, easy, efficient display of 3D graphics.
The combination of Scott, Schmirler, and Coding Labs does not explicitly teach determining coordinates of at least three calibration points which are positioned on the component inside of an outer parameter of the component;
Pentenrieder makes obvious determining coordinates of at least three calibration points which are positioned on the component inside of an outer parameter of the component; ([Figure 5] Shows an overview of translating physical calibration marker positions into model-space positions [Figure 10] Shows an example calibration marker setup with [Figure 11] Showing the scene overlaid with aligned model geometry. As can be seen, there are more than 3 markers and they are not placed only at the perimeter of the space. [Page 3 Col 2 Par 5 – Page 4 Col 1 Par 1] “Roivis was created as a marker based AR software with a graphical user interface tailored to the AR-based planning process. The usage of a marker based tracking offers an easy interface where only a printed marker and a camera are needed to start planning with the software. The crucial aspects related to the acquisition of the planning input data are the accurate positioning of the marker (which will be discussed in more detail in section 5) and the generation of applicable image data. The image data must allow successful tracking (sufficient lighting conditions and fully visible markers) and has to show the important views of the scene regarding the planning results. Given the necessary input data, planning with Roivis can be realized in a few steps. The image data is loaded by choosing the picture source. 3D models in VRML 2.0 format can be added to the scene and bound directly to a marker or to a coordinate system defined relatively to a marker using a so called offset configuration. Files for camera calibration and marker configuration data are loaded to complete the scene. The former contains the camera resolution and information on the intrinsic camera parameters. Whereas the latter defines the markers, which are used for the planning process: their size, identification number and possible offsets to reference the marker to an external coordinate system (e.g. a part coordinate system). Finally, manipulation tools are available to move and rotate the 3D models if needed.”) [Figure 6] Shows an example of wireframe virtual geometry matched to corresponding physical geometry using at least 3 calibration markers/points)
PNG
media_image1.png
577
434
media_image1.png
Greyscale
PNG
media_image2.png
397
535
media_image2.png
Greyscale
PNG
media_image3.png
451
527
media_image3.png
Greyscale
PNG
media_image4.png
396
582
media_image4.png
Greyscale
Pentenrieder is analogous art because it is within the field of industrial augmented reality and physical-virtual component matching and tracking. It would have been obvious to one of ordinary skill in the art to combine Pentenrieder with Scott, Schmirler, and Coding Labs before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to better manage repair and manufacturing applications, particularly by enabling more accurate tracking. Pentenrieder notes how tracking error is the one of the biggest causes of frustration in the field and one of the most significant factors hindering widespread adoption of such industrial augmented reality technology. ([Page 2 Col 1 Par 3-5] “Input and influence factors Most of the demonstrator applications and prototypes of AR-based industrial applications did not achieve the goal of becoming a well-accepted and often used solution in the everyday product life cycle. Possible reasons for this lack of approval can be found in [22]. Here, Regenbrecht et al. state that the maturity of contributing technologies (tracking, dis plays, content generation, wearable computing, etc.) does not suit the demanding industrial conditions yet regarding robustness, reliability, quality and practical experience. Other than the factors mentioned above, influencing elements like usability and ergonomics need to be considered for a successful industrial AR application. There are many studies and analysis on the quality of different components of AR-systems such as tracking analysis [27], [19] or display evaluations [1], [15]. One large concern is the accuracy of the overall system representing the preciseness of the virtual overlay onto the users field of view. For statements on the overall system accuracy, several influence factors need to be considered. Holloway lists tracking, calibration and modeling as main error sources in Augmented Reality systems [12]. Approaches for the combination of different sources of error and error propagation can also be found. For instance in Coelho et al. [4], a scene graph with uncertain transforms is presented, whose sources of error are also based on the work of Holloway.”) To this end, Pentenrieder presents a method for more accurate tracking between physical objects and 3D models in industrial contexts. ([Page 3 Col 2 Par 4] “Roivis was created as a marker based AR software with a graphical user interface tailored to the AR-based planning process. The usage of a marker based tracking offers an easy interface where only a printed marker and a camera are needed to start planning with the software. The crucial aspects related to the acquisition of the planning input data are the accurate positioning of the marker (which will be discussed in more detail in section 5) and the generation of applicable image data. The image data must allow successful tracking (sufficient lighting conditions and fully visible markers) and has to show the important views of the scene regarding the planning results.”) Overall, one of ordinary skill in the art would have recognized that combining Pentenrieder with Scott, Schmirler, and Coding Labs would result in a system that is significantly more accurate.
Claim 6. Scott teaches wherein the acceptance criterion or criteria are displayed ([Par 95] “In this illustrative example, selecting a graphical indicator can result in the display of additional information about nonconformances. For example, selection of the starburst 434 for nonperformance location 414 results in the display of nonconformance data 440 in window 442. Nonconformance data 440 can take a number of different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance, a three point location of the nonconformance, a work order, an image, links to addition information, or other suitable information.”) ([Par 92] “The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances.”)
Schmirler makes obvious displaying according to a static position. ([Par 89] “In the example view depicted in FIG. 8, production statistic icons 802 are rendered at a fixed location above the production area”)
Claim 7. Scott teaches wherein the acceptance criterion or criteria displayed ([Par 95] “In this illustrative example, selecting a graphical indicator can result in the display of additional information about nonconformances. For example, selection of the starburst 434 for nonperformance location 414 results in the display of nonconformance data 440 in window 442. Nonconformance data 440 can take a number of different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance, a three point location of the nonconformance, a work order, an image, links to addition information, or other suitable information.”) vary according to a real time position of the movable pointer ([Par 72-73] “As depicted, the display of at least one of points 234, nonconformance data 210, or other information for nonconformance locations 208 that are visible to portable computing device 214 at position 248 is performed using graphical indicators 257. A graphical indicator in graphical indicators 257 can include at least one of an icon, a pictogram, an ideogram, a graphic, an image, text, animation, bolding, a graph, a line, an arrow, or other suitable graphic … As human operator 228 moves with portable computing device 214, position 248 of portable computing device 214 changes. This change in position 248 can change which surfaces are visible, which can affect the display of nonconformance data 210. In this manner, the augmented view of physical object 204 can change dynamically as position 248 of portable computing device 214 changes with respect to physical object 204.”)
Coding labs makes obvious points in the second reference frame. (Page 5 Par 1 – Page 6 Par 2] “With all the objects at the right place we now need to project them to the screen. This is usually done in two steps. The first step moves all the object in another space called the View Space. The second step performs the actual projection using the projection matrix … How do we calculate the transformation matrix for View Space? Now, if you imagine you want to put the camera in World Space you would use a transformation matrix that is located where the camera is and is oriented so that the Z axis is looking to the camera target. The inverse of this transformation, if applied to all the objects in World Space, would move the entire world into View Space. Notice that we can combine the two transformations Model To World and World to View into a single transformation Model To View…. The scene is now in the most friendly space possible for a projection, the View Space. All we have to do now is to project it onto the imaginary screen of the camera. Before flattening the image, we still have to move into another, final space, the Projection Space. This space is a cuboid which dimensions are between -1 and 1 for every axis. This space is very handy for clipping (anything outside the 1:-1 range is outside the camera view area) and simplifies the flattening operation (we just need to drop the z value to get a flat image). To go from the View Space into the Projection Space we need another matrix, the View to Projection matrix, and the values of this matrix depend on what type of projection we want to perform. The two most used projections are the Orthographic Projection and the Perspective Projection. To do the Orthographic projection we have to define the size of the area that the camera can see. This is usually defined with a width and height values for the x and y axis, and a near and far z values for the z axis … Given these values we can create the transformation matrix that remaps the box area into the cuboid. The matrix that follows is transforms vectors from View Space into Ortho Projected Space and assumes a right handed coordinates system.”)
Claim 8. Scott teaches wherein at least one image of the real component is acquired, this image showing the pointer pointing to a non-conformance together with at least one acceptance criterion of the criterion area in which the virtual point associated with the pointer is located. ([Par 35] “As depicted, tablet computer 108 includes a sensor system that is configured to generate scan data describing points in space for fuselage section 102. The sensor system can include at least one of a camera, a stereo camera, a laser scanner, or some other suitable type of device that can generate scan data.” [Par 41-42] “In this illustrative example, server computer 110 can plot points (which may be retrieved from a nonconformance database, for example) that are within the coordinate cube corresponding to nonconformance locations on fuselage section 102. Server computer 110 can also determine which points for the nonconformance locations are visible in the image of fuselage section 102 displayed on tablet computer 108. These determined visible points are displayed in the image to augment the view seen by human operator 104. … Thus, tablet computer 108 provides an augmented reality view to human operator 104 that enables human operator 104 to visualize locations where nonconformances are present in fuselage section 102.” [Par 89] “With reference now to FIG. 4, an illustration of an image of a flight deck augmented to indicate nonconformance locations is depicted in accordance with an illustrative embodiment.” [Par 95] “In this illustrative example, selecting a graphical indicator can result in the display of additional information about nonconformances. For example, selection of the starburst 434 for nonperformance location 414 results in the display of nonconformance data 440 in window 442. Nonconformance data 440 can take a number of different forms. For example, nonconformance data 440 can include at least one of a description of the nonconformance, instructions for resolving the nonconformance, a three point location of the nonconformance, a work order, an image, links to addition information, or other suitable information.”)
(2) Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Scott (US 20200118345 A1) in view of Schmirler (US 20180131907 A1) in further view of Coding Labs - Article - World, View and Projection Transformation Matrices (Hereinafter Coding Labs) as well as Augmented Reality-based factory planning- an application tailored to industrial needs (Hereinafter Pentenrieder) and Matthews (US 6057827 A)
Claim 2. Scott teaches wherein each criterion area comprises at least one set of virtual points ([Par 61] “In this example, modeling application 220 can generate model 244 of the region 240 of physical object 204 in defined coordinate cube 246 using point cloud representation 242 for region 240 of physical object 204.”)([Par 92] “The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances. In this illustrative example, nonconformances that are visible in image 400 are augmented with graphical indicators while nonconformances that are obscured in image 400 are not identified using graphical indicators.”) ([Par 61] “In this example, modeling application 220 can generate model 244 of the region 240 of physical object 204 in defined coordinate cube 246 using point cloud representation 242 for region 240 of physical object 204.”) ([Par 61] “In this example, modeling application 220 can generate model 244 of the region 240 of physical object 204 in defined coordinate cube 246 using point cloud representation 242 for region 240 of physical object 204.”)
Schmirler makes obvious wherein each criterion area comprises elements to which each acceptance criterion of the criterion area applies, ([Par 152] “rendering component 308 send workflow presentation data 1602 to each recipient's wearable appliance 206. In an example scenario, rendering component 308 can render the workflow presentation as an augmented reality presentation that renders a sequence of instructions as an overlay on the user's field of view. These presentations can include graphical indicator overlays workflow, alphanumeric instructions that inform the user of the next step to be performed, feedback graphics that indicate when the step has been correctly performed or when the user has deviated from the proper workflow sequence, and other such information.) and wherein, during the step of determining whether the virtual point belongs to a criterion area, ([Par 152] “In an example scenario, rendering component 308 can render the workflow presentation as an augmented reality presentation that renders a sequence of instructions as an overlay on the user's field of view. These presentations can include graphical indicator overlays that point to or otherwise visually identify devices, workstations, or machine components that the user's attention should be focused on during a current step of the workflow, alphanumeric instructions that inform the user of the next step to be performed, feedback graphics that indicate when the step has been correctly performed or when the user has deviated from the proper workflow sequence, and other such information.” [Par 155] “Other user data 1704 may include device orientation data identifying a current orientation of the user's wearable appliance 206, which, when combined with the user's current location data, can indicate whether the user is currently viewing a correct area of the automation system for completion of a pending workflow step”)
The combination of Scott, Schmirler, Coding Labs, and Pentenrieder does not explicitly teach the coordinates in the first reference frame of the virtual point associated with the pointer are compared with the coordinates of other points in order to determine whether this virtual point belongs to one of the sets of virtual points.
Matthews makes obvious the coordinates in the first reference frame of the virtual point associated with the pointer are compared with the coordinates of other points in order to determine whether this virtual point belongs to one of the sets of virtual points. ([Col 1 line 52-59] “In another embodiment of the invention, the cursor is positioned in three dimensions by computing the coordinates of a three dimensional point on an object surface which coincides in two dimensions with the coordinates of the pointer. The three dimensional coordinates of the pointer are then reset to the three dimensional coordinates of the object point.” [Abstract] “In a computer graphics modeling system, a method for accurately positioning a cursor on a geometric point of a three dimensional object (such as an object vertex) shown in a two dimensional projected view. The system monitors the position of a movable pointer on a two dimensional projected view of the three dimensional object. When the user desires to accurately position the cursor on the object, the user signals the system via an input device or other means. The system receives the signal, which indicates that the pointer is preliminarily aligned with a geometric point on the object. In response to the signal, the system determines the position of a nearest geometric point to the pointer and moves the pointer to the spatial coordinates of the point's position. To aid in identifying geometric points, they are visibly different in the projected view from non-geometric points. In another embodiment, the cursor is positioned in three dimensions by computing the coordinates of a three dimensional point on an object surface which coincides in two dimensions with the coordinates of the pointer. The three dimensional coordinates of the pointer are then reset to the three dimensional coordinates of the object point.”)
Matthews is analogous art because it is within the field of 3D rendering. It would have been obvious to one of ordinary skill in the art to combine it with Scott, Schmirler, Coding Labs, and Pentenrieder before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to better enable identification of nonconformances, as well as easing model-side changes that may be desired related to detected nonconformances. While the combination of Scott, Schmirler, Coding Labs, and Pentenrieder describes an efficient method for detecting nonconformances, an operator may decide that a part or article of manufacture may benefit from being redesigned, perhaps to reduce chances of nonconformances being introduced in the manufacturing process. Matthews notes the loss in modelling efficiency common when precise position of tools are required, for example if a small section of a very large, complex model frequently produces nonconformances ([Col 1 line 19 -27] “Construction of three-dimensional computer models has become a valuable part of the design, construction, and manufacturing process in many fields. The efficiency of the modeling process has been restricted by the difficulty of accurately positioning modeling tools and model objects in three-dimensional space. This invention provides a set of methods for fast, accurate positioning of tools and objects in model space. These methods provide a novel economy of steps to accomplish key functions.”) To this end, Matthews presents a system that allows for efficient, very accurate positioning for 3D modelling tools ([Col 1 line 28-53] “The space-jump, which is one aspect of the invention, provides instant rendezvous of a three dimensional (3D) pointer with a distant location in space. The dynamic surface pointer, another aspect of the invention, automatically positions a 3D pointer in space relative to active objects. This automatic positioning provides very efficient performance of certain modeling tasks, including but not limited to cutting, 3D image-positioning, and feature-based or traditional boolean operations. In accordance with the invention, therefore, a method is shown and described for accurately positioning a cursor on a geometric point of a three dimensional object (such as an object vertex) shown in a two dimensional projected view. The system monitors the position of the movable pointer on a two dimensional projected view of the three dimensional object. When the user desires to accurately position the cursor on the object, the user signals the system via an input device or other means. The system receives the signal, which indicates that the pointer is preliminarily aligned with a geometric point on the object. In response to the signal, the system determines the position of a nearest geometric point to the pointer and moves the pointer to the spatial coordinates of the point's position. To aid in identifying geometric points, they are visibly different in the projected view from non-geometric points.”) Overall, one of ordinary skill in the art would have recognized that combining Matthews with Scott, Schmirler, Coding Labs, and Pentenrieder would result in a system that allowed the efficient correction of nonconformances, both through the correction of physical structures and preemptively through the modification of 3D models used in the manufacturing process.
(3) Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Scott (US 20200118345 A1) in view of Schmirler (US 20180131907 A1) in further view of Coding Labs - Article - World, View and Projection Transformation Matrices (Hereinafter Coding Labs) as well as Augmented Reality-based factory planning- an application tailored to industrial needs (Hereinafter Pentenrieder) and Gould (US 20200134877 A1)
Claim 4. Scott teaches wherein the coordinates of each real calibration point ([Par 63-65] “In one illustrative example, in localizing portable computing device 214 with model 244, localization application 222 locates location target 250 for physical object 204. Localization application 222, uses the target position 252 of location target 250 to localize portable computing device 214 with model 244 of the region 240 of physical object 204 in defined coordinate cube 246 and identifies position 248 of portable computing device 214 with respect to physical object 204. In this illustrative example, location target 250 is a physical feature. Location target 250 can be, for example, a plate, a barcode, a radio frequency identifier (RFID) device, an anchor, a spatial anchor, anchor plates, or some other suitable structure. Location target 250 is in a known location on physical object 204 and has a corresponding location in model 244 representing physical object 204. The location is in three dimensions and can be target position 252 if an orientation of location target 250 is also known. The position of location target 250 can be correlated with the corresponding location in model 244 to identify the position 248 of portable computing device 214 with respect to physical object 204.”) ([Par 92] “The live view of flight deck 402 in image 400 is augmented with graphical indicators to indicate the presence of nonconformances.”)
Coding Labs makes obvious points in the second reference frame; (Page 5 Par 1 – Page 6 Par 2] “With all the objects at the right place we now need to project them to the screen. This is usually done in two steps. The first step moves all the object in another space called the View Space. The second step performs the actual projection using the projection matrix … How do we calculate the transformation matrix for View Space? Now, if you imagine you want to put the camera in World Space you would use a transformation matrix that is located where the camera is and is oriented so that the Z axis is looking to the camera target. The inverse of this transformation, if applied to all the objects in World Space, would move the entire world into View Space. Notice that we can combine the two transformations Model To World and World to View into a single transformation Model To View…. The scene is now in the most friendly space possible for a projection, the View Space. All we have to do now is to project it onto the imaginary screen of the camera. Before flattening the image, we still have to move into another, final space, the Projection Space. This space is a cuboid which dimensions are between -1 and 1 for every axis. This space is very handy for clipping (anything outside the 1:-1 range is outside the camera view area) and simplifies the flattening operation (we just need to drop the z value to get a flat image). To go from the View Space into the Projection Space we need another matrix, the View to Projection matrix, and the values of this matrix depend on what type of projection we want to perform. The two most used projections are the Orthographic Projection and the Perspective Projection. To do the Orthographic projection we have to define the size of the area that the camera can see. This is usually defined with a width and height values for the x and y axis, and a near and far z values for the z axis … Given these values we can create the transformation matrix that remaps the box area into the cuboid. The matrix that follows is transforms vectors from View Space into Ortho Projected Space and assumes a right handed coordinates system.”)
The combination of Scott, Schmirler, Coding Labs, and Pentenrieder does not explicitly teach targeting each point in turn with a targeting element.
Gould makes obvious targeting each point in turn with a targeting element. ([Par 66-69] “In one embodiment, drill bit tip 82 is calibrated by performing several steps. First, tool 72 is calibrated to an environment, such as room R. As a result of the calibration, 3-D position 52, including orientation, of tool 72 within room R is known by system 40. … First, the worker moves the drill bit tip 82 to a floor corner at the intersection between target wall TW upon which artificial frame of reference 100 is arranged, sidewall W and floor F. The worker signals to system 40 that drill bit tip 82 is at first calibration point 102. The worker can signal to system by, for example, toggling a button on tool 72. First calibration point 102 is the origin of the Cartesian-style grid. The worker then moves along target wall TW, places the drill bit tip 82 at a corner between target wall TW and floor F, and signals to system 40 that drill bit tip 82 is at second calibration point 104. System 40 combines calibration points 102 and 104 to identify an axis (e.g., the X-coordinate axis) of a Cartesian-style grid. In one embodiment, system 40 assumes by default that target wall TW is plumb or nearly plumb. Alternatively, the worker may identify third calibration point 106 in the upper corner between target wall TW, side wall W and ceiling C and/or fourth calibration point 108 in the opposite corner between target wall TW, side wall W and ceiling C. Given three calculation points 102, 104 and 106, system 10 can calculate the orientation of target wall TW, such as whether it is plumb.”)
Gould is analogous art because it is within the field of augmented-reality enabled manufacturing. It would have been obvious to one of ordinary skill in the art to combine it with Scott, Schmirler, Coding Labs, and Pentenrieder before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to better guide correcting nonconformances. As noted by Gould, manual construction/manufacturing work can suffer from a lack of precision ([Par 2] “Construction work generally requires non-digital hand tools such as tape measures, levels, and power tools that do not calculate their location. A worker refers to a construction plan, such as a blueprint, to identify work that needs to be performed.”) One of ordinary skill in the art would have recognized that this lack of precision would also affect certain nonconformance correction procedures that require manual tooling. To this end, Gould presents a system using augmented reality that allows workers to perform tasks extremely precisely by integrating spatially-aware tools ([Par 6-7] “An augmented reality (AR) headset receives the augmented data and generates an augmented image to display to the worker wearing the AR headset. The augmented image includes a combination of the physical image in front of the AR headset and the worker, and a virtual image of objects from a construction plan. Images of the virtual objects are placed at locations in the augmented image that correspond to where the corresponding physical objects should be placed according to the construction plan. In one embodiment, the location of a spatially-aware drill is compared against a map of delicate or dangerous objects, such as an electrical wire or water pipe that should be protected. The location of the protected object is based on a construction plan, or it may be based on an as-built model that reflects where object was actually installed. A distance between the drill bit tip and the protected object is monitored and compared to a threshold. When the distance is less than the threshold, power to the drill is disabled and the drill generates a signal to display to the worker why the drill's power was disabled.”) Overall, one of ordinary skill in the art would have recognized that combining Gould with Scott, Schmirler, Coding Labs, and Pentenrieder would further improve the accuracy of the operations required to correct nonconformances.
Claim 5. Schmirler teaches wherein a representation of the virtual component is displayed in superimposition on the real component. ([Par 52-53] “ For users that are physically located on the plant floor, the VR/AR presentation system can provide automation system data, notifications, and proactive guidance to the user via modification of the user's view of his or her immediate surroundings. Such modifications can include, for example, superimposing data values or indicators on a user's view of a machine or automation system through the user's wearable computer (or other client device capable of rendering a substantially real-time view of the machine or system). The system can customize presentation of this information based on the user's role, location, line of sight, type of wearable device, and/or other contextual information. In general, the presentation system can obtain “real world” images of an industrial automation device having at least one object via a wearable appliance having at least one image sensory input. The system complements the real-world images on the appliance with virtual or augmented reality images, data, and the like that are associated with at least one identified object of the industrial automation system. The real world industrial automation device or the at least one identified object can be displayed on the appliance together with a virtual/augmented attribute display of the real world industrial automation device or the at least one object. The virtual or augmented reality presentations can include, but are not limited to, … location of controller/equipment, maps, manuals, instructions, line diagrams, ladder programs, locations, avatars, filtered views, cameras, x-ray views, removable views, troubleshooting, how-to's, error proofing, safety robots, customer information, equipment information, filters, line of sight filters, knowledge sharing portals, work flows, view/grab HMI's, line of sight (including distant line of sight), super power line of sight, authentication, privilege control, and asset tracking.” [Par 110] “Also, in some embodiments, presentation system 302 can detect when a user is viewing a control cabinet (or a virtual control cabinet) through the wearable appliance 206, and present a virtual x-ray view of the contents of the control cabinet, including a virtual view of the panel-mounted devices within the control cabinet (e.g., industrial controllers, motor drives, contactors, etc.). The virtual view can include graphical representations of the devices, as well as relevant data associated with one or more of the devices contained within the cabinet”)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael P Mirabito whose telephone number is (703)756-1494. The examiner can normally be reached M-F 10:30 am - 6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached at (571) 272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.P.M./Examiner, Art Unit 2187
/JOHN E JOHANSEN/Examiner, Art Unit 2187
Prosecution Insights