DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-6 and 10-20 are pending.
Claims 7-9 are canceled.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 10-11 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feldman et al (EP4238714A1) in view of Shu et al (US20230281785A1).
Regarding claim 10, Feldman teaches a defect depth estimation system comprising:
a training system configured to:
repeatedly receive a plurality of training image sets, each training image set comprising a first type of image having a first image format and capturing a target object having a defect, a second type of image having a second image format different from the first image format and capturing the target object having the defect, the second image data providing ground truth data indicating an actual depth of the defect,
(Feldman, Fig. 2; "For training a machine learning model 204 ... a simulator 201 produces realistic RGB and perfect depth data (representing depth measurements) of a target (training) scene ... supplemented with ground truth labels", [0046]; Shu, Fig. 8; "system 800 includes at least an image acquisition module 801, a defect segmentation module 802, and a defect determination module 803.", [0072]; Fig. 4; "acquiring a three-dimensional (3D) picture of the object to be detected; ... includes information about a defect depth of a segmented defect region", [0008]; Feldman teaches a training system receiving image sets with RGB and perfect depth data acting as ground truth; Shu teaches acquiring 3D pictures of objects capturing defects with depth information; together Feldman and Shu teach receiving training image sets of different formats capturing defects and providing actual ground truth depths)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Shu into the system or method of Feldman in order to focus the multi-format training sets onto identifying physical defects and their depths. The combination of Feldman and Shu also teaches other enhanced capabilities.
The combination of Feldman and Shu further teaches:
wherein the first image format defines a first domain and the second image format defines a second domain different from the first domain such that the difference between the first domain and the second domain defines a domain gap;
(Feldman, Fig. 2; "In this context, there is the issue of the so-called domain gap ... synthetic depth maps (i.e. depth images) are typically perfect (in terms of being complete and correct), but in reality, depth cameras have many issues", [0036]; Shu, Fig. 4; "results of 2D and 3D detection can be fused", [0008]; Feldman teaches images representing different domains that create a domain gap between perfect and real data; Shu teaches fusing results from different image formats; together Feldman and Shu teach image formats defining different domains such that their differences define a domain gap; incorporating Feldman's domain gap concept into Shu's multi-format image processing handles the discrepancies between ideal training imagery and real-world defect captures)
perform at least one domain adaption technique on the first and second images that transforms the first domain and the second domain into a target third domain that reduces the domain gap; and
(Feldman, Fig. 2; "The training data generation system then applies depth (modification) processing 202 to the synthetic depth data resulting in degraded or sparsified depth data", [0046]; "Post-processing synthetic depth maps with the modifications reduces the synthetic-real domain gap", [0042]; Shu, Fig. 7; "performing pixel-level alignment on the 2D picture and the 3D picture by using a coordinate transformation matrix", [0009]; Feldman teaches performing data modification to reduce the domain gap; Shu teaches applying transformations to align different image data; together Feldman and Shu teach performing domain adaptation techniques on the images to transform them into a target domain that reduces the domain gap; incorporating Feldman's domain gap reduction into Shu's pipeline bridges the gap between simulated and real defect imagery before model training)
train a machine learning model to learn the actual depth of the defect using the first and second images having the target third domain to generate a trained machine learning model;
(Feldman, Fig. 2; "The results of the depth data processing 202 and the RGB processing 203 are supplied to the machine-learning model 204 for training.", [0046]; Shu, Fig. 4; "inputting the acquired 2D picture to a trained defect segmentation model to obtain a segmented 2D defect mask", [0006]; "obtains a depth-related model result for actual needs, making detection of defects with a depth more accurate.", [0049]; Feldman teaches training a machine learning model using the processed domain-adapted data; Shu teaches training a model for defect segmentation and depth detection; together Feldman and Shu teach training a machine learning model to learn actual defect depths using the adapted domain images; incorporating Shu into Feldman configures the domain-adapted training process to specifically learn and predict physical depths of defects)
an image sensor configured to generate at least one 2D test image of a test object existing in real space and having a defect with a depth; and
(Feldman, Fig. 1; "takes pictures (images) of the plug 113 and socket 114 by means of cameras 115, 116", [0031]; Shu, Fig. 8; "acquire a two-dimensional (2D) picture of an object to be detected", [0015]; "Usually, a 3D camera (for example, a depth camera) is used to acquire a 3D image of the object to be detected", [0056]; Feldman teaches cameras generating test images of real-space objects; Shu teaches acquiring a 2D picture of an object to be detected for defects; together Feldman and Shu teach an image sensor generating a 2D test image of a physical test object having a defect; incorporating Shu into Feldman configures the imaging hardware specifically for capturing the defects of test objects)
a processing system configured to input the at least one 2D test image in the first image format, process the at least one 2D image to identify and isolate a localized defect region containing the defect, inputting the isolated localized defect region to the trained machine learning model and to output estimated depth information corresponding to the localized defect region of the 2D test image and indicating an estimation of the depth of the defect.
(Feldman, Fig. 1; "The controller may for example use the result of such a processing task to identify and/or locate the object 113", [0032]; Shu, Fig. 8; "input the acquired 2D picture to a trained defect segmentation model to obtain a segmented 2D defect mask ... the 2D defect mask includes information about a defect type, a defect size, and a defect location of a segmented defect region.", [0072]; "a type, an area size, and a depth of the defect can be directly determined through the result.", [0056]; Feldman teaches a controller utilizing model results to locate objects; Shu teaches processing an image through a trained model to obtain a localized defect mask and determining the defect's depth; together Feldman and Shu teach a processing system that isolates a localized defect region and outputs estimated depth information using the trained model; incorporating Shu into Feldman enables the system to evaluate real-time images, isolate exact defect masks, and accurately estimate the defect depth)
Regarding claim 11, the combination of Feldman and Shu teaches its/their respective base claim(s).
The combination further teaches the defect depth estimation system comprising of claim 10, wherein the at least one 2D test image includes a 2D image frame included in a video stream captured by the image sensor.
(Feldman, Fig. 2; "Various embodiments may (synthetically) generate and use image data (i.e. digital images) from various visual sensors (cameras) such as video, radar, LiDAR, ultrasonic, thermal imaging, motion, sonar etc.", [0060]; Shu, Fig. 8; "Usually, a 3D camera (for example, a depth camera) is used to acquire a 3D image of the object to be detected", [0056]; Feldman teaches visual sensors generating digital images from video; Shu teaches using a camera to acquire test images of the object to be detected; together Feldman and Shu teach the test image including a 2D image frame in a video stream captured by an image sensor; incorporating Feldman's video sensor modalities into Shu's defect detection system enables dynamic stream-based capture of test objects)
Regarding claim 14, the combination of Feldman and Shu teaches its/their respective base claim(s).
The combination further teaches the defect depth estimation system of claim 10, wherein the estimated depth information includes at least one of an estimated depth scalar value of the actual depth and an estimated depth map of the actual depth.
(Feldman, Fig. 2; "resulting in degraded or sparsified depth data, e.g. in form of a depth map", [0046]; "Dropping out a depth value (i.e. a pixel value of a depth image) means replacing the depth value by an invalid value", [0050]; Shu, Fig. 4; "the acquired 3D image can be stored in the form of a depth image, and a grayscale value of the depth image represents depth information in a Z direction.", [0056]; Feldman teaches providing depth data in the form of a depth map comprised of pixel depth values; Shu teaches representing defect depth information as a scalar grayscale value in a Z direction; together Feldman and Shu teach the estimated depth information including an estimated depth scalar value and an estimated depth map; incorporating Shu's Z-direction scalar value approach with Feldman's depth map processing provides a robust volumetric estimation of the physical defect)
Claim(s) 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feldman et al (EP4238714A1) in view of Shu et al (US20230281785A1) and further in view of Hubert et al (US20220245785A1) and Guizilini et al (US20220392083A1).
Regarding claim 12, the combination of Feldman and Shu teaches its/their respective base claim(s).
The combination does not expressly disclose but Hubert and Guizilini teach the defect depth estimation system comprising of claim 10,
wherein the at least one 2D test image includes a video stream containing movement of the test object, and
(Hubert, "The “video image” is a continuous sequence of “frames”", [0021]; "the individual engine blades of the engine stage can be moved successively through the image region of the video borescope.", [0024]; Hubert teaches a video stream capturing the continuous movement of a test object)
wherein the processing system performs optical flow processing on the video stream to determine the estimated depth information of the defect.
(Guizilini, "process a pair of temporally adjacent monocular image frames using a first neural network structure to produce a first optical flow estimate.", [0004]; "The optical flow estimate 140 is used to generate a depth map via triangulation.", [0023]; Guizilini teaches applying optical flow processing on successive frames to calculate depth maps, and it would be obvious to incorporate Guizilini's optical flow depth generation into Hubert's video stream analysis to systematically estimate internal defect depths without complex 3D sensors)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine Guizilini's optical flow-based depth estimation with Hubert's moving video stream analysis within the Feldman/Shu defect detection system. This combination allows the processing system to systematically calculate the internal depth information of a tracked defect directly from standard 2D video frames, eliminating the need for complex 3D sensors while retaining the highly accurate 3D defect segmentation and detection benefits of Feldman and Shu. The combination of Feldman, Shu, Hubert and Guizilini also teaches other enhanced capabilities.
Regarding claim 13, the combination of Feldman, Shu, Hubert and Guizilini teaches its/their respective base claim(s).
The combination further teaches defect depth estimation system comprising of claim 12, wherein the optical flow processing includes:
comparing a first image frame included in the 2D video stream to a second image frame of the 2D video stream that precedes the first frame;
(Hubert, "a possible movement of the engine blades can be detected by comparing in each case two successive frames.", [0027]; Guizilini, " Optical flow estimation module 315 ... to process a pair of temporally adjacent monocular image frames", [0045]; Hubert teaches comparing sequential frames; Guizilini teaches analyzing temporally adjacent frames for optical flow, rendering the limitation obvious)
determining a change in a position of the defect as the second image frame transitions to the first image frame; and
(Guizilini, "For every pixel xi t=[xi t,yi t,1]T in frame t, its predicted optical flow corresponds to the displacement ôi between frames It and It+1", [0023]; Guizilini teaches determining pixel position change/displacement between transitions of consecutive frames)
determining the estimation of the depth based on the change in the position.
(Guizilini, "From this correspondence between pixels, relative rotation Rt t+1, translation tt t+1 between frames, and camera intrinsics K, optical flow triangulation operator 145 calculates a triangulated depth map", [0023]; Guizilini teaches computing a depth map based on the determined pixel displacement/change in position; applying Guizilini's displacement-based depth calculation to Hubert's sequential video frames would be obvious to accurately reconstruct 3D defect geometries from monocular 2D inputs)
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feldman et al (EP4238714A1) in view of Shu et al (US20230281785A1) and further in view of Hubert et al (US20220245785A1).
Regarding claim 15, the combination of Feldman and Shu teaches its/their respective base claim(s).
The combination of Feldman, Shu and Hubert further teaches the defect depth estimation system of claim 10, wherein the image sensor is a borescope.
(Hubert, "A “video borescope” is a borescope which provides a continuous analog or digital video image of the image region of the borescope for further processing.", [0020]; Hubert teaches using a video borescope as the image sensor; implementing the 2D image sensor as a borescope taught by Hubert into the system of Feldman and Shu would allow non-destructive inspection of internal or hard-to-reach defect regions)
Allowable Subject Matter
Claim(s) 1-6 and 16-20 is/are allowed.
Statement of reasons for the indication of allowable subject matter: applicant’s amendment and argument filed on 1/6/2026 are persuasive.
Response to Arguments
Applicant's arguments filed on 5/12/2026 with respect to one or more of the pending claims have been fully considered but are moot in view of the new ground(s) of rejection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 7/11/2026
Prosecution Insights