DETAILED ACTION
This final rejection is responsive to the amendment filed 30 March 2026. Claims 1-20 are pending. Claims 1, 11, and 16 are independent claims. Claims 1, 11, and 16 are amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
35 U.S.C. 103
Applicant’s prior art arguments have been fully considered and they are persuasive.
Applicant argues (pgs. 8-10) that the prior art does not teach the newly amended claims which specify hidden layers.
Examiner agrees. Accordingly, a new reference, Cella (US 2022/0197306 A1), has been added to the rejection, as further detailed below.
The foregoing applies to all independent claims and their dependent claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Bakhshmand (US 2024/0160194 A1) hereinafter known as Bakhshmand in view of Mitarai (US 2018/0373738 A1) hereinafter known as Mitarai in view of Luk-Zilberman (US 2021/0027620 A1) hereinafter known as Luk-Zilberman in view of Cella (US 2022/0197306 A1) hereinafter known as Cella.
Regarding independent claim 1, Bakhshmand teaches:
receiving, by a processor, design information for a custom part; (Bakhshmand: Fig. 5 and ¶[0024] and ¶[0193]-¶[0194]; Bakhshmand further teaches receiving a golden sample image which is used as a reference image representing a clean image of the article. ¶[0308]-¶[0309] further teaches a processor.)
extracting, by the processor, feature information of the custom part from the design information; (Bakhshmand: Fig. 5 and ¶[0201]; Bakhshmand further teaches generating a feature map from the golden sample image.)
receiving, by the processor, images of the custom part in production from a recording in near real time; (Bakhshmand: Fig. 5 and ¶[0148]-¶[0149]; Bakhshmand teaches receiving an inspection image from a camera in real-time.)
...
...
An embodiment of Bakhshmand does not explicitly teach but another embodiment teaches:
concurrently inputting into a machine learning model both the recording of the images in production and a reference ... in order to verify, using machine learning, that features in the images of the custom part in production are in compliance with the feature information of the custom part from the design information ... (Bakhshmand: Fig. 5 and ¶[0253]; Bakhshmand teaches an anomaly detection module in the pipeline 500. ¶[0257] further teaches the anomaly detection module receiving the golden sample output and detecting differences between the inspection image and the golden sample image. ¶[0260]-¶[0263] further teaches identifying defects bases on the analysis and flagging defects. ¶[0027] and ¶[0161] further teaches using a machine learning, CNN, as the object detection model.)
causing an alert to be output on a user device in response to the machine learning model determining that the custom part fails to be in compliance, based on the recording of images in production and the reference video of the previous production being concurrently processed by the connected layer. (Bakhshmand: ¶[0128] and ¶[0227]; Bakhshmand teaches labeling the images and identifying the differences with a bounding box and showing the data on an operator interface.)
Bakhshmand is in the same field of endeavor as the present invention, as it is directed to defect detection during manufacturing processes. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine an inspection system that receives a real-time inspection image and a golden sample image to further use the images and machine learning to verify compliance. As such, it would have been obvious to one of ordinary skill in the art to combine these teachings because the combination would allow detecting defects, as suggested by Bakhshmand: ¶[0260].
Bakhshmand does not explicitly teach but Mitarai teaches:
... video of a previous production ... (Mitarai: Fig. 8 and ¶[0084]; Mitarai teaches capturing a video during manufacturing where no problem has occurred, i.e. a normal state and using this video to compare to a current video to detect abnormalities.)
Bakhshmand and Mitarai are in the same field of endeavor as the present invention, as the references are directed to detecting defects in a manufacturing process. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to substitute the golden sample images which are used in machine learning to detect defects and present alerts to the user as taught in Bakhshmand with a video of previous production as taught in Mitarai. Bakhshmand also teaches using multiple golden sample images. (Bakhshmand: ¶[0195]) Mitarai provides the additional functionality of using a video of a previous production. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Bakhshmand to include teachings of Mitarai, because the combination would allow using time-series data, as suggested by Mitarai: ¶[0080].
Bakhshmand in view of Mitarai does not explicitly teach but Luk-Zilberman teaches:
... , wherein the recording of the images in production and the reference video of the previous production are concurrently processed by a connected layer ... in the machine learning model; (Luk-Zilberman: Fig. 4 and ¶[0055]-¶[0056]; Luk-Zilberman teaches using live input video data and training data sets and processing them in fully-connected layers.)
Luk-Zilberman is analogous to the present invention, since it is reasonably pertinent to the problem faced by the inventor, i.e. processing live and reference video using a CNN with fully-connected layers. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine a method of using a CNN to process live video and reference images as taught in Barkshmand in view of Mitarai with further processing the videos using the fully connected layer as taught in Luk-Zilberman. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Bakhshmand and Mitarai to include teachings of Luk-Zilberman, because the combination would allow using the plurality of layers for feature extraction and classification, as suggested by Luk-Zilberman: ¶[0039].
Bakhshmand in view of Mitarai does not explicitly teach but Cella teaches:
... , wherein the recording of the images in production is input into a first input layer for processing by a first plurality of hidden layers in the machine learning model, wherein the reference video of the previous production is input into a second input layers for processing by a second plurality of hidden layers in the machine learning model, ... via the first plurality of hidden layers and the second plurality of hidden layers, respectively; (Cella: Fig. 105 and ¶[1531] and ¶[1565]; Cella teaches a neural network with hidden layers for processing image data.)
Cella is analogous to the present invention, since it is reasonably pertinent to the problem faced by the inventor, i.e. processing imagery using a CNN with fully-connected layers. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine a method of using a CNN to process live video and reference images with further processing the videos using the fully connected layer as taught in as taught in Barkshmand in view of Mitarai in view of Luk-Zilberman with further utilizing hidden layers as taught in Cella. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Bakhshmand, Mitarai, and Luk-Zilberman to include teachings of Cella, because the combination would allow processing of information based on weights of the connections, as suggested by Cella: ¶[1531].
Regarding claim 2, Bakhshmand in view of Mitarai in view of Luk-Zilberman further teaches the method of claim 1.
Bakhshmand further teaches:
further comprising selecting, by the processor, reference images from the images of the custom part in production that are in compliance with the feature information of the custom part. (Bakhshmand: ¶[0151]-¶[0153]; Bakhshamnd further teaches inspection images which are labelled as “OK”.)
Regarding claim 3, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 1.
Bakhshmand further teaches:
further comprising: receiving, by the processor, the recording in near real time, wherein the recording is a video recording and is an initial video recording of the custom part in production; and Bakhshmand: Fig. 5 and ¶[0091] and ¶[0148]-¶[0149]; Bakhshmand teaches receiving an inspection image from a camera in real-time during a manufacturing/production process.)
outputting, by the processor, an alert to a computing device when a feature in the images of the custom part in production is not in compliance with the feature information of the custom part. (Bakhshmand: ¶[0128] and ¶[0227]; Bakhshmand teaches labeling the images and identifying the differences with a bounding box and showing the data on an operator interface.)
Regarding claim 4, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 1.
Bakhshmand further teaches:
wherein the machine learning comprises applying a convolutional neural network model that identifies subprocesses of the custom part in production from the images of the custom part in production from the recording. (Bakhshmand: ¶[0027] and ¶[0161]-¶[0162] and ¶[0200]; Bakhshmand further teaches using CNNs to perform an analysis on inspection images and golden sample images. ¶[0144] and ¶[0149] further teaches using real-time imaging. Accordingly, the system operates through the manufacturing process, which may be interpreted as subprocesses.)
Regarding claim 5, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 1.
Bakhshmand further teaches:
wherein the machine learning comprises applying a convolutional neural network model that determines the features in the images of the custom part in production are in compliance with an image from the feature information of the custom part. (Bakhshmand: ¶[0027] and ¶[0161]; Bakhshmand further teaches using a machine learning, CNN, as the object detection model.)
Regarding claim 6, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 4.
Bakhshmand further teaches:
wherein the convolutional neural network model is trained to receive the images of the custom part in production from the recording, the subprocesses from the reference video and reference images for each subprocess to identify plural subprocesses of the custom part in production from the images of the custom part in production from the recording. (Bakhshmand: ¶[0027] and ¶[0161]-¶[0162], ¶[0195], and ¶[0200]; Bakhshmand further teaches using CNNs to perform an analysis on inspection images and golden sample images. ¶[0144] and ¶[0149] further teaches using real-time imaging. Accordingly, the system operates through the manufacturing process, which may be interpreted as subprocesses.)
Mitarai further teaches a reference video. (Mitarai: Fig. 8 and ¶[0080]-¶[0084])
Regarding claim 7, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 5.
Bakhshmand further teaches:
wherein the convolutional neural network model is trained to determine that the features in the images of the custom part in production are in compliance using at least one image of the custom part with plural features that are not in compliance with the features of the custom part and at least one image of the custom part with the features that are in compliance with the features of the custom part. (Bakhshmand: ¶[0027] and ¶[0161]-¶[0162] and ¶[0200]; Bakhshmand further teaches using CNNs to perform an analysis on inspection images and golden sample images.)
Regarding claim 8, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 1.
Bakhshmand further teaches:
wherein the verifying using the machine learning comprises applying plural convolutional neural network models that determine the features in the images of the custom part in production are in compliance with an image from the feature information of the custom part in subprocesses of the production. (Bakhshmand: ¶[0027] and ¶[0161]-¶[0162] and ¶[0200]; Bakhshmand further teaches using CNNs to perform an analysis on inspection images and golden sample images. ¶[0144] and ¶[0149] further teaches using real-time imaging. Accordingly, the system operates through the manufacturing process.)
Regarding claim 9, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 2.
Bakhshmand further teaches:
further comprising: verifying, by the processor, using the machine learning in a subsequent production of another custom part that plural features in images of the another custom part in the subsequent production are in compliance with plural features of the reference images. (Bakhshmand: Fig. 23 and ¶[0064] and ¶[0398]; Bakhshmand further teaches using a pipeline that can take into account multiple articles. Further, nothing in Bakhshmand precludes from using the system on another article.)
Regarding claim 10, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the method of claim 1.
Bakhshmand further teaches:
further comprising: storing the recording in a storage system; and inputting images of the custom part in production from the stored recording in a convolutional neural network model that determines plural features in plural images of another custom part in a subsequent production are in compliance with plural features in the images of the custom part in production from the recording. (Bakhshmand: ¶[0128] and ¶[0227]; Bakhshmand teaches labeling the images and identifying the differences with a bounding box and showing the data on an operator interface. Figs. 2-3 and ¶[0072]-¶[0074] further teaches storing the information.)
Regarding claim 11, Bakhshmand teaches:
receive, by a processor, images of a custom part in production from an initial video recording in near real time; (Bakhshmand: Fig. 5 and ¶[0148]-¶[0149]; Bakhshmand teaches receiving an inspection image from a camera in real-time.)
concurrently input, by the processor, ... the images of the custom part in production .... into a convolutional neural network model that determines features in the images of the custom part are similar with feature information of the custom part are similar with feature information the custom part and the reference ... ; (Bakhshmand: ¶[0027] and ¶[0161]-¶[0162] and ¶[0200]; Bakhshmand further teaches using CNNs to perform an analysis on inspection images and golden sample images. ¶[0144] and ¶[0149] further teaches using real-time imaging. Accordingly, the system operates through the manufacturing process, which may be interpreted as subprocesses.)
...
...
An embodiment of Bakhshmand does not explicitly teach but another embodiment teaches:
receive, by the processor, an indication from the convolutional neural network model that a feature in the images of the custom part in production is not similar with the feature information of the custom part; and (Bakhshmand: Fig. 5 and ¶[0253]; Bakhshmand teaches an anomaly detection module in the pipeline 500. ¶[0257] further teaches the anomaly detection module receiving the golden sample output and detecting differences between the inspection image and the golden sample image. ¶[0260]-¶[0263] further teaches identifying defects bases on the analysis and flagging defects. ¶[0027] and ¶[0161] further teaches using a machine learning, CNN, as the object detection model.)
causing, by the processor, an alert to be output on a computing device in response to receiving the indication that the custom part in production is not in compliance with the feature information of the custom part. (Bakhshmand: ¶[0128] and ¶[0227]; Bakhshmand teaches labeling the images and identifying the differences with a bounding box and showing the data on an operator interface.)
Bakhshmand is in the same field of endeavor as the present invention, as it is directed to defect detection during manufacturing processes. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine an inspection system that receives a real-time inspection image and a golden sample image to further use the CNN to verify compliance. As such, it would have been obvious to one of ordinary skill in the art to combine these teachings because the combination would allow detecting defects, as suggested by Bakhshmand: ¶[0260].
Bakhshmand does not explicitly teach but Mitarai teaches:
... a reference video of a previous production ... (Mitarai: Fig. 8 and ¶[0084]; Mitarai teaches capturing a video during manufacturing where no problem has occurred, i.e. a normal state and using this video to compare to a current video to detect abnormalities.)
Bakhshmand and Mitarai are in the same field of endeavor as the present invention, as the references are directed to detecting defects in a manufacturing process. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to substitute the golden sample images which are used in machine learning to detect defects and present alerts to the user as taught in Bakhshmand with a video of previous production as taught in Mitarai. Bakhshmand also teaches using multiple golden sample images. (Bakhshmand: ¶[0195]) Mitarai provides the additional functionality of using a video of a previous production. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Bakhshmand to include teachings of Mitarai, because the combination would allow using time-series data, as suggested by Mitarai: ¶[0080].
Bakhshmand in view of Mitarai does not explicitly teach but Luk-Zilberman teaches:
... , wherein the images of the custom part in production and the reference video of the previous production are concurrently processed by a connected layer in the machine learning model; (Luk-Zilberman: Fig. 4 and ¶[0055]-¶[0056]; Luk-Zilberman teaches using live input video data and training data sets and processing them in fully-connected layers.)
Luk-Zilberman is analogous to the present invention, since it is reasonably pertinent to the problem faced by the inventor, i.e. processing live and reference video using a CNN with fully-connected layers. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine a method of using a CNN to process live video and reference images as taught in Barkshmand in view of Mitarai with further processing the videos using the fully connected layer as taught in Luk-Zilberman. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Bakhshmand and Mitarai to include teachings of Luk-Zilberman, because the combination would allow using the plurality of layers for feature extraction and classification, as suggested by Luk-Zilberman: ¶[0039].
Bakhshmand in view of Mitarai does not explicitly teach but Cella teaches:
... , wherein the recording of the images in production is input into a first input layer for processing by a first plurality of hidden layers in the convolutional neural network model, wherein the reference video of the previous production is input into a second input layers for processing by a second plurality of hidden layers in the convolutional neural network model, ... via the first plurality of hidden layers and the second plurality of hidden layers, respectively; (Cella: Fig. 105 and ¶[1531] and ¶[1565]; Cella teaches a neural network with hidden layers for processing image data.)
Cella is analogous to the present invention, since it is reasonably pertinent to the problem faced by the inventor, i.e. processing imagery using a CNN with fully-connected layers. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine a method of using a CNN to process live video and reference images with further processing the videos using the fully connected layer as taught in as taught in Barkshmand in view of Mitarai in view of Luk-Zilberman with further utilizing hidden layers as taught in Cella. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Bakhshmand, Mitarai, and Luk-Zilberman to include teachings of Cella, because the combination would allow processing of information based on weights of the connections, as suggested by Cella: ¶[1531].
Regarding claim 12, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the computer program product of claim 11.
Bakhshmand further teaches:
wherein the program instructions are further executable to: receive, by the processor, design information for the custom part; and extract, by the processor, the feature information of the custom part from the design information. (Bakhshmand: Fig. 5 and ¶[0024] and ¶[0193]-¶[0194]; Bakhshmand further teaches receiving a golden sample image which is used as a reference image representing a clean image of the article. ¶[0308]-¶[0309] further teaches a processor. Fig. 5 and ¶[0201] further teach generating a feature map from the golden sample image.)
Regarding claim 13, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the computer program product of claim 11.
Bakhshmand further teaches:
wherein the convolutional neural network model is trained to receive the images of the custom part in production from the initial video recording, subprocesses of the production of the custom part from the reference video and reference images for each subprocess to identify plural subprocesses of the custom part in production from the images of the custom part in production from the initial video recording. (Bakhshmand: ¶[0027] and ¶[0161]-¶[0162] and ¶[0200]; Bakhshmand further teaches using CNNs to perform an analysis on inspection images and golden sample images. ¶[0144] and ¶[0149] further teaches using real-time imaging. Accordingly, the system operates through the manufacturing process, which may be interpreted as subprocesses.)
Mitarai further teaches a reference video. (Mitarai: Fig. 8 and ¶[0080]-¶[0084])
Regarding claim 14, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the computer program product of claim 11.
Bakhshmand further teaches:
wherein the convolutional neural network model is trained to receive the images of the custom part in production from the initial video recording and subprocesses of the production of the custom part from a reference video to determine the features in the images of the custom part are similar with the feature information of the custom part. (Bakhshmand: ¶[0027] and ¶[0161]-¶[0162] and ¶[0200]; Bakhshmand further teaches using CNNs to perform an analysis on inspection images and golden sample images.)
Regarding claim 15, Bakhshmand in view of Mitarai in view of Luk-Zilberman in view of Cella further teaches the computer program product of claim 11.
Bakhshmand further teaches:
wherein the program instructions are further executable to: store the initial video recording in a storage system; and input images of the custom part in the production from the stored initial video recording in a convolutional neural network model that determines plural features in plural images of another custom part in a subsequent production are in compliance with plural features in the images of the custom part in production from the initial video recording. (Bakhshmand: ¶[0128] and ¶[0227]; Bakhshmand teaches labeling the images and identifying the differences with a bounding box and showing the data on an operator interface. Figs. 2-3 and ¶[0072]-¶[0074] further teaches storing the information.)
Regarding claims 16-20, these claims recite a system that performs the method of claims 11-15; therefore, the same rationale for rejection applies.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEX OLSHANNIKOV whose telephone number is (571)270-0667. The examiner can normally be reached M-F 9:30-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott Baderman can be reached at 571-272-3644. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEKSEY OLSHANNIKOV/Primary Examiner, Art Unit 2118
Prosecution Insights