DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/24/2024 was filed and is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 2, 4-6, 10, 12-14, 18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ziabari (“Simurgh: A Framework for CAD-Driven Deep Learning Based X-Ray CT Reconstruction”, 2022).
Regarding claims 1 and 10, Ziabari teaches A non-destructive evaluation (NDE) terminal (Ziabari, pg 3863, Section 1, lines 1-3: “Industrial cone beam X-ray computed tomography (XCT) has been used as a key tool for non-destructive characterization (NDC) of 3D-printed/additively manufactured (AM) parts”) comprising processing circuitry configured to:
receive image (Ziabari, pg 3864, Section 2, reproduced below:
PNG
media_image1.png
291
641
media_image1.png
Greyscale
. “sparse-view analytic reconstruction from a few 3D printed samples corresponding to the CAD model that were scanned using XCT”) and modeling data for a part or assembly (Ziabari, see Section 2 image above, “CAD model of the part of interest”) from multiple sources (Ziabari, see Section 2, multiple sources include XCT, CAD; further, defect library and defect simulator (Section 2.2));
employ a machine learning model (Ziabari, see Section 2 image above, “Simurgh framework” is being interpreted as involving a machine learning model) to fuse the image and modeling data (Ziabari, pg 3865, column 1, paragraph before Section 2.3: “We bootstrapped defects from the library and simulated defects and embedded them into the CAD models before simulations for training and testing”. “Embedded” is being interpreted as involving the fusion process and creates the fused image data. Also see figure 1 for an overview of the framework) into tabular data (Ziabari, pg 3866, Figure 5c, Comparison with pycnometry; top-right plot, which shows Defect Density [from pycnometry] which is being interpreted to involve “tabular data” that allows for graphical plots to be created) and fused image data associating predicted stress values (Ziabari, pg 3866, Figure 5c, Comparison with pycnometry; top-right plot, defect density is being interpreted as “predicted stress values”) with respective locations of the part or assembly (Ziabari, pg 3864, Figure 3 text, “Green arrows identifies the flaws detected via segmentation, and are also present in HR data”. Which shows the location of the detect flaws); and
link the tabular data and fused image data together for display (Ziabari, Figures 3 and Figure 5c show the tabular data and fused image data are displayed together).
Regarding claim 2, Ziabari teaches The NDE terminal of claim 1, wherein the image and modeling data comprises X-ray computed tomography (XCT) image data (Ziabari, pg 3864, column 2, lines 2-5: “sparse-view analytic reconstruction from a few 3D printed samples corresponding to the CAD model that were scanned using XCT,”), baseline finite element analysis data (Ziabari, pg 3865, column 1, lines 3-5: “We then used the simulator developed in [12] to simulate pores, cracks and other types of defects.” The defect simulator is being interpreted as involving finite element analysis), and computer aided design (CAD) model data (Ziabari, pg 3864, column 2, lines 1-2: “The overall inputs to the framework are the CAD model of the part of interest”).
Regarding claim 4, Ziabari teaches The NDE terminal of claim 1, wherein employing the machine learning model comprises employing supervised training (Ziabari, see Section 2.4 image below, “training data” shows “supervised training”) and inference including backpropagation (Ziabari, see Section 2.4 image below, “U-Net architecture” and “loss function” are being interpreted to involve backpropagation to update the weights of the neural network as part of the inference) via a convolutional neural network (CNN) (Ziabari, pg 3865, column 2, Section 2.4, last 2 paragraphs:
PNG
media_image2.png
444
644
media_image2.png
Greyscale
. “2.5D U-Net architecture” is being interpreted as involving a “convolutional neural network”, supported by the “convolutional layers”) to generate the tabular data (Ziabari, pg 3866, Figure 5c, Comparison with pycnometry; top-right plot, which shows Defect Density [from pycnometry] which is being interpreted to involve “tabular data” that allows for graphical plots to be created) and fused image data (Ziabari, pg 3865, column 1, paragraph before Section 2.3: “We bootstrapped defects from the library and simulated defects and embedded them into the CAD models before simulations for training and testing”. “Embedded” is being interpreted as involving the fusion process and creates the fused image data. Also see figure 1 for an overview of the framework).
Regarding claim 5, Ziabari teaches The NDE terminal of claim 1, wherein linking the tabular data and fused image data together for display comprises providing a plurality of instances of a linking element (Ziabari, pg 3866, column 2, paragraph before Section 4, reproduced below:
PNG
media_image3.png
578
642
media_image3.png
Greyscale
. The plot in Figure 5c and the fused image data from the Simurgh framework in Figure 5 show tabular and fused image data, respectively. The “segmented pores and defects” and “pyncnometry data”, or defect density, show that they are linked; which is interpreted as involving a linking element),
each one of the plurality of instances of the linking element linking an entry in the tabular data (Ziabari, pg 3866, Figure 5c, the plot shows tabular data was used to create the plot) to a portion of the fused image data (Ziabari, see nearest image above, “segmented pores and defects in the reconstruction volumes” is being interpreted to involve “a portion of the fused image”) associated with a corresponding one of the respective locations of the part or assembly (Ziabari, see nearest image above, “segmented pores and defects in the reconstruction volumes” is being interpreted to involve locations of the part or assembly. As one with ordinary skill in the art would know, segmentation has the location. See “pore” and “crack” defects in Figure 2 as having a location).
Regarding claim 6, Ziabari teaches The NDE terminal of claim 5, wherein the processing circuitry is further configured to display the fused image data at a user interface (Ziabari, pg 3866, Figure 5, which shows a display of fused images and defect data, that is being interpreted to involve using a user interface) with contours in the fused image data (Ziabari, see pg 3866 image in claim 5, “segmented pores and defects in the reconstruction volumes” are being interpreted as “contours” in the fused image data) correlating to respective different levels of the predicted stress values (Ziabari, pg 3864, Figure 2 text and images, which shows “pore and crack” that are being interpreted as “different levels of the predicted stress values”).
Regarding claim 18, Ziabari teaches The method of claim 10, wherein the image and modeling data comprises NDE image data from multiple sources (Ziabari, Figure 1, which shows multiple sources such as Real X-Ray CT measurement, X-ray CT simulator), and a two dimensional or three dimensional model of the part or assembly (Ziabari, Figure 1, which shows CAD, which may be 2D or 3D model of the part or assembly, as one with ordinary skill in the art would know).
Claim 12 is rejected using the same rationale as applied to claim 4 discussed above.
Claim 13 is rejected using the same rationale as applied to claim 5 discussed above.
Claim 14 is rejected using the same rationale as applied to claim 6 discussed above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3, 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ziabari, in view of Colin (US 2016/0264262 A1, 2013).
Regarding claim 3, Ziabari teaches The NDE terminal of claim 2, wherein the image and modeling data (Ziabari, pg 3864, Section 2, reproduced below:
PNG
media_image1.png
291
641
media_image1.png
Greyscale
. XCT and CAD model, respectively) further comprises
Ziabari does not appear to specifically teach “other sensor data including at least one of a group of options comprising: ultrasound image data, conventional image data from a visual camera, infrared (IR) image data from an IR camera, and temperature data from a pyrometer”
Pertaining to the same field of endeavor, Colin teaches
other sensor data (Colin, [0146]: “Such non-destructive testing means may employ probes…or any other type of probe capable of conducting a local examination of the material in a particular zone”) including at least one of a group of options comprising:
ultrasound image data (Colin, [0146]: “ultrasound probes”),
conventional image data from a visual camera (Colin, [0047]: “the viewing means 13, a digital camera in the example illustrated”),
infrared (IR) image data from an IR camera (Colin, [0146]: “thermal camera”), and
temperature data from a pyrometer (Colin, [0146]: “temperature measurement means”).
Ziabari and Colin are considered to be analogous art because they are directed to non-destructive evaluation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for non-destructive evaluation (as taught by Ziabari) to include various sensor data such as ultrasound (as taught by Colin) because the combination provides an improvement to anomaly detection (Colin, [0007]).
Claim 11 is rejected using the same rationale as applied to claim 3 discussed above.
Claim(s) 7, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ziabari, in view of Jiang (US 2020/0287123 A1, 2020).
Regarding claim 7, Ziabari teaches The NDE terminal of claim 6, wherein the contours are one of color contours and shade contours (Jiang, see Jiang image below along with Figure 3A, which shows color contours [a version of gray] and shade [the various intensities of gray]), and one or more of the color or shade contours correspond to predicted stress higher than a threshold value (Jiang, [0007-0008], reproduced below:
PNG
media_image4.png
268
1000
media_image4.png
Greyscale
; Figures 3A and 3B, which show various shades or color contours indicating various stress concentrations; because of the various shades, this shows a threshold was met for the stress to show up in the image as a contour).
Ziabari and Jiang are considered to be analogous art because they are directed to non-destructive evaluation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for non-destructive evaluation (as taught by Ziabari) to include a contour map for stress distribution (as taught by Jiang) because the combination provides an improvement to non-destructive methods that are non-invasive (Jiang, 0066-0067). Further, contour maps, or stress distribution maps are a common method to visualize stress distribution, as one with ordinary skill in the art would know.
Claim 15 is rejected using the same rationale as applied to claim 3 discussed above.
Claim(s) 8, 9, 16, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ziabari, in view of Hao (“EISeg: An Efficient Interactive Segmentation Tool based on PaddlePaddle”, 2022).
Regarding claim 8, Ziabari teaches The NDE terminal of claim 5, wherein the processing circuitry is further configured to display an interactive table based on the tabular data simultaneously with the fused image data (Hao, Figure 1, reproduced below:
PNG
media_image5.png
924
1168
media_image5.png
Greyscale
. Examiner recommends viewing the color image instead. The table near the top-right that contains “sheep 1” and “sheep 2” is being interpreted as tabular and interactive), each entry in the interactive table being selectable (Hao, see image above, the various sheep objects are clickable on the table) to highlight a corresponding one of the respective locations (Hao, see image above, clicking on the sheep objects in the table proceeds to highlight the respective sheep object at the location) in the fused image data via a corresponding one of the instances of the linking element (Hao, see image, the fused image data are the sheep shown. The action of clicking the sheep objects in the table is linked to the segmented sheep in the image).
Ziabari and Hao are considered to be analogous art because they are directed to image segmentation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for non-destructive evaluation with annotation and segmentation (as taught by Ziabari, Section 2.2) to include an interactive table that highlights the segmented region (as taught by Hao) because the combination provides an improvement to image segmentation and annotation (Hao, abstract). Further, interactive image annotation with segmentation would be common knowledge to one with ordinary skill in the art at the time of filing; otherwise, the user would not be able to edit the segmentation nor the table.
Regarding claim 9, Ziabari teaches The NDE terminal of claim 5, wherein the processing circuitry is further configured to display an interactive table based on the tabular data (Hao, Figure 1, reproduced below:
PNG
media_image5.png
924
1168
media_image5.png
Greyscale
. Examiner recommends viewing the color image instead. The table near the top-right that contains “sheep 1” and “sheep 2” is being interpreted as tabular and interactive) simultaneously with the fused image data (Hao, see image above, the segmented sheep are shown simultaneously with the table near the top-right), each one of the respective locations in the fused image data being selectable (Hao, see image above, the segmented sheep on the left image are part of the interactive nature of the image annotation tool) to highlight a corresponding entry in the interactive table (Hao, see image above, Examiner recommends viewing the color version of the image as sheep2 in the table is highlight while the user works on the respective sheep segmentation) via a corresponding one of the instances of the linking element (Hao, see image above, the segmented sheep is linked with the label in the table).
Ziabari and Hao are considered to be analogous art because they are directed to image segmentation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for non-destructive evaluation with annotation and segmentation (as taught by Ziabari, Section 2.2) to include an interactive table that highlights the segmented region (as taught by Hao) because the combination provides an improvement to image segmentation and annotation (Hao, abstract). Further, interactive image annotation with segmentation would be common knowledge to one with ordinary skill in the art at the time of filing; otherwise, the user would not be able to edit the segmentation, the table, or know which segmentation belong to which label in the table.
Claim 16 is rejected using the same rationale as applied to claim 8 discussed above.
Claim 17 is rejected using the same rationale as applied to claim 9 discussed above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Torbali et al ("A State-of-the-Art Review of Non-Destructive Testing Image Fusion and Critical Insights on the Inspection of Aerospace Composites towards Sustainable Maintenance Repair Operations”, Feb 2023) discloses non-destructive testing methods that include image fusion of multiple sources.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY B DUONG whose telephone number is (571)272-1358. The examiner can normally be reached Monday - Thursday 10a-9p (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.B.D./Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights