DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicants Amendments filed on May 4, 2026, has been entered and made of record.
Currently pending Claim(s): 1-20
Independent Claim(s): 1, 19
Amended Claim(s): 1, 4, 7-8, 15 19-20
Canceled Claim(s): 21-25
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description:
The “display” 2410 (discussed in paragraph [0161]) is not shown on Fig. 20.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
In paragraph [0073], reference number ‘218’ should read ‘216’. Reference number ‘218’ refers to a fixing element, while the specification refers to the connector ‘216’.
In paragraph [0074], delete the extra indent before “3.”.
Remove the empty paragraph [0093], and renumber following paragraphs accordingly.
Appropriate correction is required.
Claim Interpretation – 35 U.S.C 112(f)
In view of Applicant’s amendments to Claim 19, Claim 19 and its dependent claims are no longer being interpreted under 112(f).
Response to Arguments
This office action is responsive to the Applicant’s Arguments/Remarks Made in an Amendment
received on May 4, 2026.
Claim Rejections – 35 U.S.C 101
In view of Applicant’s amendments to Claim 1 and Claim 19, and Applicant’s arguments (on Remarks pg. 7-8), the 101 rejection of Claims 1-15 and 19-20 is withdrawn.
Claim Rejections – Double Patenting
Originally, (in the claim set dated April 26, 2024) Claims 1-20 were rejected on the grounds of nonstarter double patenting over Sessinghaus et al. (U.S. Patent No 11, 978,197).
The applicant has currently amended the claims to include multiple computer components. However, after reviewing the amended claims and conducting a new search, the Examiner argues that the amended claims are not patentably distinct over Sessinghaus and Kitajima et al. (US Pub No 20200151844).
As to Claim 1, Sessinghaus teaches recognizing least one inspection part in the inspection image and at least one reference part in the reference data representing the reference object (see Claim 1, lines 6-8, “recognizing at least one inspection part in the inspection image and at least one reference part in the reference image”):
registering the inspection image relative to the reference data by transforming image data of the inspection image using the at least one inspection part and the at least one reference part to align the inspection image with the reference image and generate registration data (see Claim 1, lines 9-10, “registering the inspection image onto the reference image using the inspection part and the reference part and providing a set of registration data”, and the Examiner notes that ‘registering’ two images inherently comprises aligning images);
and detecting, by the computing unit, at least one assembly error of the inspection object using the inspection image, the reference data, and the registration data (see Claim 1, lines 11-12, “and checking for at least one error using the inspection image, the reference image, and the set of registration data”).
The Examiner notes that the instant application now recites the term ‘assembly error’, instead of simply ‘error’ claimed in the U.S. Patent. However, the Examiner notes that in Claim 4 of the instant application, ‘assembly error’ is defined as “at least one of an incorrect part error, a part orientation error, an alignment error, a fixing element error, or a measurement error”. In Claim 4 of the previously issued patent, the term ‘error’ is defined in the exact same manner. Thus, the Examiner has found no patentable distinction between the terms ‘error’ and ‘assembly error’.
Claims 2-18 and 20 of the instant application and are almost identical to that of the earlier granted patent. Claim 2 of the U.S. patent recites “wherein at least one of the inspection part” and “at least one reference part”. However, Claim 2 of the instant application recites “wherein the at least one inspection part and the at least one reference part correspond to each other”. Thus, the instant application is merely broadening the scope of the initial claim so that multiple parts of the inspection image and the reference image may be registered and compared.
Claims 3, 7, and 19 are similarly altered in the same way. The U.S. Patent recites “the inspection part” and the “the reference part”, while the instant application recites “the at least one inspection part” and “the at least one reference part.
Claims 4, 7, 9, 14, 15, 19 and 20 have been amended to recite ‘assembly error’ instead of simply ‘error’. However, as stated previously, the Examiner has found no patentable distinction between the terms ‘error’ and ‘assembly error’.
Sessinghaus fails to recite a camera system, inspection data bus, reference data bus, computing unit comprising a microprocessor, program code memory and data memory, and a display unit.
However in an analogous art, Kitajima teaches a machine vision inspection method (see paragraph [0010], “According to one aspect of the present invention, an image processing apparatus is configured to process an image…The comparator is configured to compare an input inspection image with the plurality of models”), comprising:
providing a machine vision inspection system (see paragraph [0044], “Alternatively, the pattern matching performed by the image processing apparatus 300 may be performed for, for example, inspection of individuals of the workpieces”)
a camera system (see Fig. 3, camera 500),
an inspection data bus coupled to the camera system (see Fig. 3, bus 310, connected to camera 500)
a reference data bus (see Fig. 3 bus 310; the Examiner notes that the instant specification states that the ‘inspection data bus’ and the ‘reference data bus’ may be encompassed by the same bus (see paragraph [00153] of the instant specification , “The inspection data interface 2002 and / or the reference data interface 2004 may be implemented as a wired or a wireless data bus. They may be implemented either as separate physical entities or as one and the same physical entity”)),
a reference data memory coupled to the reference data bus (see external storage device 800 connected to bus 310):
a computing unit coupled to the inspection data bus and the reference data bus (see Fig. 3 image processing apparatus) ,
the computing unit including a microprocessor unit (see Fig. 3, processing unit 301, and see paragraph [0105], “The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU))”),
a program code memory (see Fig. 2, ROM 302),
and a data memory (see Fig. 2, RAM 303);
and a display unit coupled to the computing unit (see Fig. 3, display apparatus 700):
acquiring, by the camera system, an inspection image of an inspection object (see paragraph [0035], “a camera 500 serving as an image capturing apparatus”):
transmitting the inspection image to the computing unit via the inspection data bus (see paragraph [0054], “The CPU 301 obtains an image signal from the camera 500 via the bus 310 and the interface 312.”):
storing, by the reference data memory, reference data including a reference image representing a reference object (see paragraph [0049], “this reference image may be stored in advance in a storage device such as the HDD 304 or the external storage device 800,”);
and transmitting the reference data to the computing unit via the reference data bus (see paragraph [0049], “the CPU 301 obtains the data of reference image from the storage device such as the HDD 304 or the external storage device 800.”)
recognizing, by the computing unit, at least one inspection part in the inspection image and at least one reference part in the reference data representing the reference object (see paragraph [0039], “In a manufacturing line constituted as illustrated in FIG. 2, pattern matching is performed to detect or measure an actual position or orientation of the entirety or a part of the workpieces W1 and W2, a specific portion of the robot arm 402, or the like. Therefore, in this pattern matching, a model for matching a specific portion of the workpieces W1 and W2, the robot arm 402, or the like is used”, and the model is interpreted as the reference representing the reference object)
registering, by the computing unit, the inspection image relative to the reference data (see paragraph [0013], “see paragraph [0067], “The provisional model generation portion is configured to generate a plurality of provisional models by for pattern matching using a reference image in which a target object of the pattern matching is recorded. The matching score obtaining portion is configured to perform the pattern matching between each of the plurality of provisional models and each of a plurality of evaluation images in which the target object of the pattern matching is recorded”, where the Examiner has interpreted ‘pattern matching’ as registering)
and detecting, by the computing unit, at least one assembly error (see paragraph [0044], “Alternatively, the pattern matching performed by the image processing apparatus 300 may be performed for, for example, inspection of individuals of the workpieces W1 and W2 or an assembly in which the workpieces W1 and W2 are integrated”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the inspection system taught by Kitajima with the inspection method taught by Sessinghaus. The motivation for doing so would be to would be to increase the accuracy of image registration and defect detection. Kitajima teaches in paragraph [0007], “In addition, pattern matching of this kind is sometimes performed on an image obtained by imaging a workpiece or a manufacturing apparatus in a manufacturing line of a product, and results thereof are sometimes used for monitoring, manufacture control and the like. In the case where erroneous detection occurs in such a system, there is a possibility that a defective product is manufactured or breakdown of a manufacturing apparatus occurs due to erroneous operation thereof. Therefore, a method and a system that generate a model with which erroneous detection is unlikely to occur and reliable pattern matching can be performed are desired”. Thus, it would have been obvious to combine the method taught by Zhao with the with the system disclosed by Sessinghaus in order to obtain the invention as claimed in Claim 1.
Thus, for the reasons stated above, the Examiner maintains the double patenting rejection of Claims 1-20.
Claim Rejections – 35 U.S.C 102 and 103
In view of amendments filed on, the Applicant has amended independent Claim 1 to recite multiple additional computer components . Originally, (in the claim set dated April 26, 2024) Claim 1 recited, “An inspection method, comprising: comparing an inspection image of an inspection object to a reference image of a reference object; recognizing at least one inspection part in the inspection image and at least one reference part in the reference image, wherein the at least one inspection part and the at least one reference part correspond to each other; registering the inspection image onto the reference image using the at least one inspection part and the at least one reference part and providing a set of registration data, and checking for at least one error using the inspection image, the reference image, and the set of registration data” and was rejected by Zhao et al. (Zhao, Gang et al., “A mask R-CNN based method for inspecting cable brackets in aircraft”, 2020). Claim 19 was rejected by Zhao in view of Uchida (US Pub No 2019/0333204).
As discussed in the following paragraphs, the combination of Zhao and Uchida does not render obvious the newly amended claims. However, the Applicant’s amendment necessitated the new grounds of rejection presented in this Office action. Upon conducting a new search, the Examiner argues that the newly amended Claims 1 and 19 are unpatentable over Zhao and Kitajima, since Kitajima teaches the claimed computing system.
In view of Applicant’s Arguments/Remarks filed on , with respect to the claims, the Applicant explained (on Remarks pg. 9, paragraph 3 – pg. 10, paragraph 1 that Zhao and Uchida fails to teach the newly amended claim. The applicant explained that Zhao, “does not disclose the amended machine vision system”. The Examiner agrees. Zhao fails to explicitly teach a reference data bus, reference data memory, and computing unit. The applicant then argued that Uchida fails to cure the deficiencies of Zhao. The applicant explained that Uchida “does not teach or suggest the claimed registration-based workflow in which the computing unit transforms image data of the inspection image using the at least one inspection part and the at least one reference part to align the inspection image with the reference image and generate registration data. Nor does Uchida teach or suggest detecting at least one assembly error of the inspection object using the inspection image, the reference data, and the registration data.”
The Examiner respectfully disagrees with the Applicant’s assertion. Uchida teaches generating registration data (see paragraph [0065], “Next, in S602, the frame selected in S601 is analyzed and a reference object defined by the inspection content definition information 113 is recognized. In S602, the position of the work object can be determined through template matching with reference to a template”, where the Examiner has interpreted ‘template matching’ to be registration) through a computing unit (see Fig. 1A CPU 101). Additionally, Uchida teaches a reference data bus (see Fig 1A), a reference data memory (see Fig. 1A, storage unit 104), program code memory (see Fig. 1A, ROM 101), and data memory (Fig. 1A, RAM 103). However, Uchida fails to explicitly teach the claimed microprocessor and a bus coupled to the camera system.
The Examiner also previously identified Dal Mutto et al. (US Pub No 20200372625), which teaches a method of inspecting an object, which comprises an inspection data bus coupled to a camera system (see Fig. 2). The applicant argued “even if Dal Mutto were considered for a camera, processor, memory, or bus arrangement, Dal Mutto does not teach or suggest the claimed registration-based inspection workflow.” The Examiner agrees that Dal Mutto fails to explicitly teach registering reference image data with inspection data. Dal Mutto teaches comparing reference image data with inspection image data (see paragraph [0048]), but fails to explicitly teach that the image data is registered. However, after completing an updated search, the Examiner identified Kitajima et al. (US Pub No 20200151844), which teaches a registration based workflow, an inspection data bus connected to a camera system, a microprocessor, and the other added computer components.
Thus, the Applicant’s amendment necessitated the new grounds of rejection presented in Office Action, and the independent claims 1 and 19 are rejected under 35 USC 103 as being unpatentable over Zhao and Kitajima. Therefore, the rejection to the dependent Claims 2-18 and 20 are maintained.
Double Patenting
A rejection based on double patenting of the “same invention” type finds its support in the language of 35 U.S.C. 101 which states that “whoever invents or discovers any new and useful process... may obtain a patent therefor...” (Emphasis added). Thus, the term “same invention,” in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957).
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over Claims 1-20 of Sessinghaus et al. (U.S. Patent No. 11,978,197), hereinafter Sessinghaus, in view of Kitajima et al. (US Pub No 20200151844), hereinafter Kitajima.
Although the claims at issue are not identical, they are not patentably distinct from each other. Sessinghaus teaches recognizing least one inspection part in the inspection image and at least one reference part in the reference data representing the reference object (see Claim 1, lines 6-8, “recognizing at least one inspection part in the inspection image and at least one reference part in the reference image”):
registering the inspection image relative to the reference data by transforming image data of the inspection image using the at least one inspection part and the at least one reference part to align the inspection image with the reference image and generate registration data (see Claim 1, lines 9-10, “registering the inspection image onto the reference image using the inspection part and the reference part and providing a set of registration data”, and the Examiner notes that ‘registering’ two images inherently comprises aligning images);
and detecting, by the computing unit, at least one assembly error of the inspection object using the inspection image, the reference data, and the registration data (see Claim 1, lines 11-12, “and checking for at least one error using the inspection image, the reference image, and the set of registration data”).
The Examiner notes that the instant application now recites the term ‘assembly error’, instead of simply ‘error’ claimed in the U.S. Patent. However, the Examiner notes that in Claim 4 of the instant application, ‘assembly error’ is defined as “at least one of an incorrect part error, a part orientation error, an alignment error, a fixing element error, or a measurement error”. In Claim 4 of the previously issued patent, the term ‘error’ is defined in the exact same manner. Thus, the Examiner has found no patentable distinction between the terms ‘error’ and ‘assembly error’.
Claims 2-18 and 20 of the instant application and are almost identical to that of the earlier granted patent. Claim 2 of the U.S. patent recites “wherein at least one of the inspection part” and “at least one reference part”. However, Claim 2 of the instant application recites “wherein the at least one inspection part and the at least one reference part correspond to each other”. Thus, the instant application is merely broadening the scope of the initial claim so that multiple parts of the inspection image and the reference image may be registered and compared.
Claims 3, 7, and 19 are similarly altered in the same way. The U.S. Patent recites “the inspection part” and the “the reference part”, while the instant application recites “the at least one inspection part” and “the at least one reference part.
Claims 4, 7, 9, 14, 15, 19 and 20 have been amended to recite ‘assembly error’ instead of simply ‘error’. However, as stated previously, the Examiner has found no patentable distinction between the terms ‘error’ and ‘assembly error’.
Claim 9 of the U.S. Patent recites “searching for presence of at least one characteristic property”. However, the instant application recites “searching for presence of at least one characteristic property, wherein the at least one characteristic property”. Thus, the application is merely broadening the scope of the initial claim so that multiple characteristic properties can be searched. Claim 10 has similarly been broadened to state “at least one characteristic property” instead “the characteristic property”.
Sessinghaus fails to recite a camera system, inspection data bus, reference data bus, computing unit comprising a microprocessor, program code memory and data memory, and a display unit.
However in an analogous art, Kitajima teaches a machine vision inspection method (see paragraph [0010], “According to one aspect of the present invention, an image processing apparatus is configured to process an image…The comparator is configured to compare an input inspection image with the plurality of models”), comprising:
providing a machine vision inspection system (see paragraph [0044], “Alternatively, the pattern matching performed by the image processing apparatus 300 may be performed for, for example, inspection of individuals of the workpieces”)
a camera system (see Fig. 3, camera 500),
an inspection data bus coupled to the camera system (see Fig. 3, bus 310, connected to camera 500)
a reference data bus (see Fig. 3 bus 310; the Examiner notes that the instant specification states that the ‘inspection data bus’ and the ‘reference data bus’ may be encompassed by the same bus (see paragraph [00153] of the instant specification , “The inspection data interface 2002 and / or the reference data interface 2004 may be implemented as a wired or a wireless data bus. They may be implemented either as separate physical entities or as one and the same physical entity”)),
a reference data memory coupled to the reference data bus (see external storage device 800 connected to bus 310):
a computing unit coupled to the inspection data bus and the reference data bus (see Fig. 3 image processing apparatus) ,
the computing unit including a microprocessor unit (see Fig. 3, processing unit 301, and see paragraph [0105], “The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU))”),
a program code memory (see Fig. 2, ROM 302),
and a data memory (see Fig. 2, RAM 303);
and a display unit coupled to the computing unit (see Fig. 3, display apparatus 700):
acquiring, by the camera system, an inspection image of an inspection object (see paragraph [0035], “a camera 500 serving as an image capturing apparatus”):
transmitting the inspection image to the computing unit via the inspection data bus (see paragraph [0054], “The CPU 301 obtains an image signal from the camera 500 via the bus 310 and the interface 312.”):
storing, by the reference data memory, reference data including a reference image representing a reference object (see paragraph [0049], “this reference image may be stored in advance in a storage device such as the HDD 304 or the external storage device 800,”);
and transmitting the reference data to the computing unit via the reference data bus (see paragraph [0049], “the CPU 301 obtains the data of reference image from the storage device such as the HDD 304 or the external storage device 800.”)
recognizing, by the computing unit, at least one inspection part in the inspection image and at least one reference part in the reference data representing the reference object (see paragraph [0039], “In a manufacturing line constituted as illustrated in FIG. 2, pattern matching is performed to detect or measure an actual position or orientation of the entirety or a part of the workpieces W1 and W2, a specific portion of the robot arm 402, or the like. Therefore, in this pattern matching, a model for matching a specific portion of the workpieces W1 and W2, the robot arm 402, or the like is used”, and the model is interpreted as the reference representing the reference object)
registering, by the computing unit, the inspection image relative to the reference data (see paragraph [0013], “see paragraph [0067], “The provisional model generation portion is configured to generate a plurality of provisional models by for pattern matching using a reference image in which a target object of the pattern matching is recorded. The matching score obtaining portion is configured to perform the pattern matching between each of the plurality of provisional models and each of a plurality of evaluation images in which the target object of the pattern matching is recorded”, where the Examiner has interpreted ‘pattern matching’ as registering)
and detecting, by the computing unit, at least one assembly error (see paragraph [0044], “Alternatively, the pattern matching performed by the image processing apparatus 300 may be performed for, for example, inspection of individuals of the workpieces W1 and W2 or an assembly in which the workpieces W1 and W2 are integrated”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the inspection system taught by Kitajima with the inspection method taught by Sessinghaus. The motivation for doing so would be to would be to increase the accuracy of image registration and defect detection. Kitajima teaches in paragraph [0007], “In addition, pattern matching of this kind is sometimes performed on an image obtained by imaging a workpiece or a manufacturing apparatus in a manufacturing line of a product, and results thereof are sometimes used for monitoring, manufacture control and the like. In the case where erroneous detection occurs in such a system, there is a possibility that a defective product is manufactured or breakdown of a manufacturing apparatus occurs due to erroneous operation thereof. Therefore, a method and a system that generate a model with which erroneous detection is unlikely to occur and reliable pattern matching can be performed are desired”. Thus, it would have been obvious to combine the method taught by Zhao with the with the system disclosed by Sessinghaus in order to obtain the invention as claimed in Claim 1.
All other limitations from Claims 1-20 of the instant application correspond to Claims 1-20 of the U.S. Patent. Thus, the instant application is not patentably distinct from U.S. Patent No. 11,978,197 in view of Kitajima.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-6, 9-10, 16-17, 19-20, are rejected under 35 U.S.C. 103 as being unpatentable over by Zhao et al. (Zhao, Gang et al., “A mask R-CNN based method for inspecting cable brackets in aircraft”, 2020), in view of Kitajima et al. (US Pub No 20200151844), hereinafter Kitajima.
As to Claim 1, Zhao teaches a machine vision inspection method (see pg. 2, lines 36-38, “In this work, a semi-automatic assembly states inspection method for aircraft cable brackets is proposed”), comprising:
recognizing at least one inspection part in the inspection image and at least one reference part in the reference data representing the reference object (see pg. 2, lines 43-45, “bracket recognizer based on Mask R-CNN is trained and applied to segment 45 brackets from image to be inspected”),:
registering the inspection image relative to the reference data by transforming image data of the inspection image using the at least one inspection part and the at least one reference part to align the inspection image with the reference image (see pg. 2, lines 45-4, “thirdly, image registration between the target image and the standard global image is conducted to get the corresponding standard partial image”, and see pg. 6, Fig. 7, where the inspection image and reference image are aligned),
and generate registration data (see page 10, Fig. 11 Image registration result (b), where the images are registration data):
and detecting at least one assembly error of the inspection object using the inspection image, the reference data, and the registration data (see page 7, lines 320-324, “After getting the inspection result, visualization module is 320 adopted for the convenience of inspector. For every bracket, 321 it will be labeled with various colors, such as green color for 322 correct bracket, red color for missed bracket, and yellow color 323 for incorrect bracket”, and see Fig.8, where missing an incorrect brackets are shown).
Zhao fails to explicitly teach the multiple claimed computer components. However, in an analogous art, Kitajima teaches a machine vision inspection method (see paragraph [0010], “According to one aspect of the present invention, an image processing apparatus is configured to process an image…The comparator is configured to compare an input inspection image with the plurality of models”), comprising:
providing a machine vision inspection system (see paragraph [0044], “Alternatively, the pattern matching performed by the image processing apparatus 300 may be performed for, for example, inspection of individuals of the workpieces”)
a camera system (see Fig. 3, camera 500),
an inspection data bus coupled to the camera system (see Fig. 3, bus 310, connected to camera 500)
a reference data bus (see Fig. 3 bus 310; the Examiner notes that the instant specification states that the ‘inspection data bus’ and the ‘reference data bus’ may be encompassed by the same bus (see paragraph [00153] of the instant specification , “The inspection data interface 2002 and / or the reference data interface 2004 may be implemented as a wired or a wireless data bus. They may be implemented either as separate physical entities or as one and the same physical entity”)),
a reference data memory coupled to the reference data bus (see external storage device 800 connected to bus 310):
a computing unit coupled to the inspection data bus and the reference data bus (see Fig. 3 image processing apparatus) ,
the computing unit including a microprocessor unit (see Fig. 3, processing unit 301, and see paragraph [0105], “The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU))”),
a program code memory (see Fig. 2, ROM 302),
and a data memory (see Fig. 2, RAM 303);
and a display unit coupled to the computing unit (see Fig. 3, display apparatus 700):
acquiring, by the camera system, an inspection image of an inspection object (see paragraph [0035], “a camera 500 serving as an image capturing apparatus”):
transmitting the inspection image to the computing unit via the inspection data bus (see paragraph [0054], “The CPU 301 obtains an image signal from the camera 500 via the bus 310 and the interface 312.”):
storing, by the reference data memory, reference data including a reference image representing a reference object (see paragraph [0049], “this reference image may be stored in advance in a storage device such as the HDD 304 or the external storage device 800,”);
and transmitting the reference data to the computing unit via the reference data bus (see paragraph [0049], “the CPU 301 obtains the data of reference image from the storage device such as the HDD 304 or the external storage device 800.”)
recognizing, by the computing unit, at least one inspection part in the inspection image and at least one reference part in the reference data representing the reference object (see paragraph [0039], “In a manufacturing line constituted as illustrated in FIG. 2, pattern matching is performed to detect or measure an actual position or orientation of the entirety or a part of the workpieces W1 and W2, a specific portion of the robot arm 402, or the like. Therefore, in this pattern matching, a model for matching a specific portion of the workpieces W1 and W2, the robot arm 402, or the like is used”, and the model is interpreted as the reference representing the reference object)
registering, by the computing unit, the inspection image relative to the reference data (see paragraph [0013], “see paragraph [0067], “The provisional model generation portion is configured to generate a plurality of provisional models by for pattern matching using a reference image in which a target object of the pattern matching is recorded. The matching score obtaining portion is configured to perform the pattern matching between each of the plurality of provisional models and each of a plurality of evaluation images in which the target object of the pattern matching is recorded”, where the Examiner has interpreted ‘pattern matching’ as registering)
and detecting, by the computing unit, at least one assembly error (see paragraph [0044], “Alternatively, the pattern matching performed by the image processing apparatus 300 may be performed for, for example, inspection of individuals of the workpieces W1 and W2 or an assembly in which the workpieces W1 and W2 are integrated”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the inspection system taught by Kitajima with the inspection method taught by Zhao. The motivation for doing so would be to would be to increase the accuracy of image registration and defect detection. Kitajima teaches in paragraph [0007], “In addition, pattern matching of this kind is sometimes performed on an image obtained by imaging a workpiece or a manufacturing apparatus in a manufacturing line of a product, and results thereof are sometimes used for monitoring, manufacture control and the like. In the case where erroneous detection occurs in such a system, there is a possibility that a defective product is manufactured or breakdown of a manufacturing apparatus occurs due to erroneous operation thereof. Therefore, a method and a system that generate a model with which erroneous detection is unlikely to occur and reliable pattern matching can be performed are desired”. Thus, it would have been obvious to combine the method taught by Zhao with the with the system disclosed by Kitajima in order to obtain the invention as claimed in Claim 1.
As to Claim 2, Zhao in view of Kitajima teaches the inspection method according to claim 1, wherein at least one of the at least one inspection part or the at least one reference part is recognized using a neural network (see Zhao, pg. 2, lines 43-45, “bracket recognizer 44 based on Mask R-CNN is trained and applied to segment 45 brackets from image to be inspected”, where CNN stands for convolutional neural network).
As to Claim 3, Zhao in view of Kitajima teaches the inspection method according to claim 1 one of the preceding claims, wherein the registering comprises: performing a homography estimation based on at least one reference base point derived from the at least one reference part and at least one inspection base point derived from the at least one inspection part (see Zhao, pg. 2, lines 214-217, “Secondly, according to the multi-scale template matching 213 result, the projection position of the target image’s four corner 214 points in the standard global image can be obtained Then, four pairs of corner points are used to calculate homography matrix H1 between target image and standard global image as follows”.
As to Claim 4, Zhao in view of Kitajima teaches that the at least one assembly error comprises at least one of an incorrect part error but fails to teach a part orientation error, an alignment error, a fixing element error, or a measurement error (see Zhao, page 7, lines 320-324, “After getting the inspection result, visualization module is 320 adopted for the convenience of inspector. For every bracket, 321 it will be labeled with various colors, such as green color for 322 correct bracket, red color for missed bracket, and yellow color 323 for incorrect bracket”, and see Fig.8, where missing an incorrect brackets are shown) .
As to Claim 5, Zhao in view of Kitajima teaches the inspection object is or corresponds to a composite construction object (see Zhao, Fig. 12, aircraft wall),
the composite construction object comprises a plurality of inspection construction parts (see Zhao, Fig. 12, aircraft wall with multiple cable brackets, where each bracket is a construction part),
the reference object is or corresponds to a reference composite construction object (see Zhao, Fig. 7, standard global image, model aircraft wall)
and the reference composite construction object comprises a plurality of reference construction parts. (see Zhao, Fig. 7, standard global image of aircraft wall with several cable brackets)
As to Claim 6, Zhao in view of Kitajima teaches the inspection method according to claim 1 ,wherein the reference image comprises at least one of BIM data, CAD data, or a set of construction parts data (see Zhao, pg.4, Fig.3(a), Rendered standard global RGB image (a), and see pg. 2, lines 112-114, “In virtue of simulation technology based on Open 112 Scene Graph (OSG), a platform is developed to automatically generate synthetic realistic images with pixel-level annotations based on 3D digital model”, where the 3D digital model is the construction parts data).
As to Claim 9, Zhao in view of Kitajima teaches the inspection method according to Claim 1, wherein detecting the at least one assembly error comprises: searching for presence of at least one characteristic property, wherein the at least one characteristic property includes a characteristic corresponding to a class of construction objects which a reference composite construction object belongs to. (see Zhao, page 13, lines 153-158, “In order to obtain the accurate instance segmentation results, a brackets recognizer based on Mask R-CNN is trained. The framework of brackets recognizer based on Mask R-CNN is illustrated in Fig. 4. It consists mainly of two parts: backbone for feature extraction and head for object detection (location and classification) and mask prediction” and see page 13, lines 38 through 50, “then, assembly states inspection for brackets is 49 executed by automatically comparing the shapes of brackets 50 in the target image with the shapes of corresponding brackets”, where the characteristic property is the shape corresponding to a bracket, and the bracket is a class of construction objects).
As to Claim 10, Zhao in view of Kitajima teaches that the at least one characteristic property is or at least comprises a horizontal or at least essentially horizontal construction part (see Zhao, Fig 2(a), the shape of some cable brackets are essentially horizontal).
As to Claim 16, Zhao in view of Kitajima teaches presenting an overlaid image containing at least an area of the inspection image and at least an area of the reference image (see Zhao, Fig. 5, Brackets inspection results, and see Fig 8, Examples of bracket matching result).
As to Claim 17, Zhao in view of Kitajima teaches the inspection method according to claim 16 one of the preceding claims, wherein the overlaid image comprises at least one error-marking label (see Zhao, page 7, lines 320-324, “After getting the inspection result, visualization module is 320 adopted for the convenience of inspector. For every bracket, 321 it will be labeled with various colors, such as green color for 322 correct bracket, red color for missed bracket, and yellow color 323 for incorrect bracket”, and see Fig. 8, red marks for Incorrect bracket and missing bracket).
As to Claim 19, Zhao in view of Kitajima teaches a machine vision system configured to inspect an inspection object comprising a plurality of inspection parts, the machine vision system comprising the same components and steps disclosed in Claim 1. Therefore, the rejection and rationale of Claim 19 are analogous to that of Claim 1.
As to Claim 20, Zhao in view of Kitajima teaches the at least one error comprises at least one of an incorrect part error but fails to teach a part orientation error, an alignment error, a fixing element error, or a measurement error (see Zhao, page 7, lines 320-324, “After getting the inspection result, visualization module is 320 adopted for the convenience of inspector. For every bracket, 321 it will be labeled with various colors, such as green color for 322 correct bracket, red color for missed bracket, and yellow color 323 for incorrect bracket”) .
Claim 7-8, and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over by Zhao et al. (Zhao, Gang et al., “A mask R-CNN based method for inspecting cable brackets in aircraft”, 2020), in view of Kitajima (US Pub No 20200151844), hereinafter Kitajima, and further in view of Uchida (US Pub No 2019/0333204), hereinafter Uchida.
As to Claim 7, Zhao in view of Kitajima fails to teach that detecting the at least one assembly error comprises: defining a focus region based on the at least one reference part, and comparing the at least one inspection part being inside the focus region to the at least one reference part. However, Uchida teaches an image processor for inspection (see abstract) where a “inspection target area” can be determined from an image (see paragraph [0027], “The inspection target selection unit 112 includes an inspection target candidate selection unit that extracts inspection target candidates from the captured images 109”) , that the target area may be extracted based on a reference object (see paragraph [0057], “In S603, the inspection target area may be extracted based on the position relative to the reference object previously defined”) and that the inspection part inside this target area can be compared to a “correct answer image”, (see paragraph [0035], “The correct answer image 209 corresponds to the inspection contents and, for example, is an image in a state where the screw has been correctly fastened. In the image inspection, it can be determined whether the assembly has been correctly performed, through comparison of the image of the inspection target area 208 and the correct answer image 209”, and see Fig 2C where “target area” is identified, and compared to “correct answer image” 209). Uchida is combinable with Zhao and Kitajima as all three are from the analogous field of image analysis for inspection. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the inspection target area taught by Uchida with the inspection method taught by Zhao in view of Kitajima. The motivation for doing so would be to determine reduce the time needed to inspect an assembled product. Uchida teaches in paragraph [0071], “it is possible to appropriately determine the inspection timing and the inspection target area of the assembled product based on the assembling work in the factory. This eliminates interrupting the work only for inspection of the assembled product”. Thus, it would have been obvious to combine the target area taught by Uchida with the method taught by Zhao in order to obtain the invention as claimed in Claim 7.
As to Claim 8, Zhao in view of Kitajima teaches checking whether the at least one inspection part has at least one of an incorrect part error or a part orientation error (see Zhao, page 13, lines 321-324, “For every bracket, 321 it will be labeled with various colors, such as green color for 322 correct bracket, red color for missed bracket, and yellow color 323 for incorrect bracket”). Zhao fails to teach that the incorrect part is found within a focus region. However, Uchida teaches that an incorrect fastened part can be identified within a target region (see paragraph [0036], “An image of an inspection target area 210 is an image in the state where the screw has been correctly fastened, and an image of an inspection target area 211 is an image in a state where one of two screws has not been correctly fastened. When the image of the inspection target area 211 is inspected, it is determined that the screw has not been correctly fastened, and a warning is notified to the worker”, where the screw is an inspection part). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the inspection target area taught by Uchida with the inspection method taught by Zhao in view of Kitajima . The motivation for doing so would be to determine reduce the time needed to inspect a target area of an assembled as taught by Uchida in paragraph [0071]. Thus, it would have been obvious to combine the target area taught by Uchida with the method taught by Zhao in view Kitajima in order to obtain the invention as claimed in Claim 8.
As to Claim 13, Zhao in view of Kitajima and Uchida teaches analyzing the focus region using a neural network (see paragraph [0092] of Uchida, “The processing by the work determination unit, the image inspection unit, etc. among the above-described processing units may be performed by a learnt model that has performed machine learning, in place of the processing units… The learnt model can be configured by, for example, a neural network model”).
As to Claim 14, Zhao in view of Kitajima fails to teach detecting the at least one assembly error further comprises searching for at least one fixing element within the focus region. However, Uchida teaches that a screw and (which is a fixing element) can be identified within a focus region (see paragraph [0036], “ An image of an inspection target area 210 is an image in the state where the screw has been correctly fastened, and an image of an inspection target area 211 is an image in a state where one of two screws has not been correctly fastened. When the image of the inspection target area 211 is inspected, it is determined that the screw has not been correctly fastened”). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the inspection target area taught by Uchida with the inspection method taught by Zhao in view of Kitajima. The motivation for doing so would be to determine reduce the time needed to inspect a target area of an assembled as taught by Uchida in paragraph [0071]. Thus, it would have been obvious to combine the target area taught by Uchida with the method taught by Zhao in view of Kitajima in order to obtain the invention as claimed in Claim 14.
As to Claim 15, Zhao in view of Kitajima teaches that detecting the at least one assembly error comprises counting fixing elements (see Zhao, pg. 8, “Nt is the number of brackets in the target image”, and the brackets are interpreted as fixing elements), but fails to teach that detecting the assembly error comprises counting fixing elements within the focus region. However, Uchida teaches that errors regarding fastening elements within a focus region can be identified (see paragraph [0036], “An image of an inspection target area 210 is an image in the state where the screw has been correctly fastened, and an image of an inspection target area 211 is an image in a state where one of two screws has not been correctly fastened. When the image of the inspection target area 211 is inspected, it is determined that the screw has not been correctly fastened”, where the inspection target area is the focus region, and the fastening element is a screw). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the inspection target area taught by Uchida with the inspection method taught by Zhao in view of Kitajima. The motivation for doing so would be to determine reduce the time needed to inspect a target area of an assembled as taught by Uchida in paragraph [0071]. Thus, it would have been obvious to combine the target area taught by Uchida with the method taught by Zhao in view of Kitajima in order to obtain the invention as claimed in Claim 8.
Claims 11- 12 are rejected under 35 U.S.C. 103 as being unpatentable over by Zhao et al. (Zhao, Gang et al., “A mask R-CNN based method for inspecting cable brackets in aircraft”, 2020) in view of Kitajima et al. (US Pub No 20200151844), hereinafter Kitajima, and further in view of Daisuke et al. (JP 2002236100), hereinafter Daisuke.
As to Claim 11, Zhao in view of Kitajima teaches that a scaling factor may be computed (see Zhao, pg. 3, Section 3.3.2., “Then, template matching between integrated target mask and integrated standard global mask with different scale is used to determine the scale and position of target image”). However, Zhao fails to teach explicitly that the scaling factor is computed with respect to an element having known dimension. Zhao also fails to explicitly searching for the element having the known dimensions in the inspection image based on the scaling factor. However, Daisuke teaches elements with known dimensions can be used to determine magnification (see paragraph [0007], “When the inspection object is imaged, rectangles, circles or straight lines whose dimensions are already known or marks in a shape (a '+' shape, an L-shape, a 'cross +' shape) as a combination of straight lines area arranged inside the same screen, so as to be fetched simultaneously. By using the marks arranged at equal intervals inside the image, the correction processing operation for magnification, position, inclination or the like is performed”, where the marks correspond to ), and searching for the element having known dimensions (see paragraph [0032], “On the monitor 72, the position of the digital camera system 200 is adjusted so that A1 and B1 and A2 and B2 substantially match, and an image of the inspection target is picked up and an image is captured, where marks B1 and B2 are marks corresponding to the element. Daisuke is combinable with Zhao because both are from the analogous field of image analysis for inspection. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the magnification corresponding to an element taught by Daisuke with the scaling and method taught by Zhao. The motivation for doing so would be to correct scaling issues caused by movement of the imaging device. Daisuke teaches in paragraph [0004], “When an image pickup device such as a TV camera is moved to an inspection location to input an image, there is a problem in that the magnification and the tilt of the image differ for each image to be picked up.” Thus, it would have been obvious to one of ordinary skill to combine the teachings of Daisuke with the teachings of Zhao and Kitajima in order to obtain the inventio as claimed in Claim 11.
As to Claim 12, Zhao fails to explicitly teach that the element having the known dimensions is or at least comprises at least one of a fiducial, a mark, or a tag. However, Daisuke teaches that a mark may be given to an element of known dimension (see paragraph [0007], “When the inspection object is imaged, rectangles, circles or straight lines whose dimensions are already known or marks in a shape (a '+' shape, an L-shape, a 'cross +' shape) as a combination of straight lines area arranged inside the same screen, so as to be fetched simultaneously’). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inventio to combine the fiducial marks taught by Daisuke with the teachings of Zhao and Kitajima. The motivation for doing so would be to correct scaling issues caused by movement of the imaging device as taught by Daisuke in paragraph [0004].
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over by Zhao et al. (Zhao, Gang et al., “A mask R-CNN based method for inspecting cable brackets in aircraft”, 2020), in view of Kitajima et al. (US Pub No 20200151844), hereinafter Kitajima, in view of in view of Daisuke et al. (JP2002236100), and further in view of Abidi et al. (Abidi, Besma R. et al., “Operator Assisted Threat Assessment for Carry-On luggage Inspection”, 2001), hereinafter Abidi.
As to Claim 18, Zhao in view Kitajima and Daisuke fails to teach modifying a visibility of at least one of the areas of the inspection image or the reference image based on information received from a sliding button. However, Abidi teaches a system for inspecting carrying luggage, which allows the user to modify the visibility of the inspection image of the carry-on (see Fig, 6, images of luggage scene with sliding bar, and see caption for Figure 6., “Sliding intensity bar allows for manual manipulation of the luggage scene by the screener….This is a case of image enhancement by intensity selection”, where the user enhanced the visibility by modifying the intensity of the image). Abidi is combinable with Zhao, Kitajima and Daisuke as all three are from the analogous field of image analysis for inspecting objects. Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the sliding intensity bar taught by Abidi with the teachings of Zhao, Kitajima, and Daisuke. The motivation for doing so would be to allow the user to more easily interpret images. Abidi teaches in on page 5, paragraph 2, “The focus is on displaying raw and gradually processed data by several methods deemed to be helpful towards data interpretation and decision making. At various levels of processing, user interfaces, displays, and visualization models were designed in an effort to ease the luggage, X-ray image interpretation task.”). Thus, it would have been obvious to one of ordinary skill to combine the sliding bar taught by Abidi with teachings of Zhao and Daisuke in order to obtain the invention as claimed in Claim 18.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hicks et al. (US Pub No 20130051611) teaches a method and system for identifying regions of interest by registering a reference image and an inspection image.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOUMYA THOMAS whose telephone number is (571)272-8639. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.T./Examiner, Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights