Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-6 and 20 are rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. that was filed in 2020 and in view of United States Patent Application Pub. No.: US20210073727A1 to Burch.
In regard of claim 1 and 20, Watson disclosed “ …1. A method for determining a condition of a component ( see abstract and claim 1 where a camera can take an image of the defective component for a pattern and then provide and out put that the part if defective )
Watson is silent but BURCH teaches “...of a remotely operated vehicle operating on a rail system of an automated storage and retrieval system for goods holders, said method comprising: selecting the component of the remotely operated vehicle,” (see claims 68-75 where the drone can provide a flight plan to provide an inspection of a component including a second drone or flight device: and see FIG. 9-11 show At 1050, based at least in part on the first and second testing image performance metrics, a recommended number of defect images for training the defect detection portion is determined. For example, the defect detection portion fits a training image performance versus number of defect images used for model training results with an appropriate curve, fits a testing image performance versus number of defect images used for model training results with an appropriate curve, and utilizes a convergence algorithm to find a value of the number of defect images used to train the model for which the testing fitted curve and the training fitted curve converge toward each other, in a manner similar to that described above in connection with FIG. 7. The method 1000 then proceeds to 1060.
At 1060, after at least the recommended number of defect images have been used to train the defect detection portion, the camera is utilized to acquire new images of workpieces during a run mode, and the defect detection portion is utilized to analyze the new images to determine defect images that include workpieces with defects (e.g., including classifying the new images a defect images or non-defect images). The method 1000 then ends.
FIG. 11 shows a flow diagram of a method 1100 for operating a machine vision inspection system similar to that of FIG. 1. For example, in a manner similar to that described above in connection with FIG. 7, a defect detection portion is trained using a plurality of groups of training images, wherein each group of training images includes a predetermined number of defect images. More particularly, the defect detection portion is trained during a series of training cycles. During the first training cycle, the defect detection portion is trained with a first group of training images. During the second training cycle, the defect detection portion is trained with the first group of training images along with a second group of training images. During the third training cycle, the defect detection portion is trained with the first and second groups of training images along with a third group of training images. During the fourth training cycle, the defect detection portion is trained with the first, second, and third groups of training images along with a fourth group of training images. After each training cycle, the accuracy of the defect detection portion is determined using two sets of images, including the set of training images used to train the defect detection portion in the previous training cycle, and a set of testing images not used to train the defect detection portion in the previous cycle. )
Watson discloses “...recording an image of the component,( see claim 1 -2 where the component can be determined to be defective and claims 3-8 where the image and slopes of the image of the part can be determined )
identifying the component based on the recorded image, and ( see FIG. 1-3 where The vision inspection machine 12 includes a moveable workpiece stage 32 and an optical imaging system 34 that may include a zoom lens or interchangeable objective lenses. The zoom lens or interchangeable objective lenses generally provide various magnifications for the images provided by the optical imaging system 34. Various exemplary implementations of the machine vision inspection system 10 are also described in commonly assigned U.S. Pat. Nos. 7,454,053; 7,324,682; 8,111,905; and 8,111,938, each of which is hereby incorporated herein by reference in its entirety.
FIG. 2 is a block diagram of a control system portion 120 and a vision components portion 200 of a machine vision inspection system 100 similar to the machine vision inspection system of FIG. 1, including certain features disclosed herein. As will be described in more detail below, the control system portion 120 is utilized to control the vision components portion 200. The control system portion 120 may be arranged to exchange data and control signals with the vision components portion 200. The vision components portion 200 includes an optical assembly portion 205, light sources 220, 230, 240, 300, and a workpiece stage 210 having a central transparent portion 212. The workpiece stage 210 is controllably movable along x- and y-axes that lie in a plane that is generally parallel to the surface of the stage where a workpiece 20 may be positioned.
The optical assembly portion 205 includes a camera system 260 and an interchangeable objective lens 250. In some implementations, the optical assembly portion 205 may optionally include a variable focal length (VFL) lens, e.g., a tunable acoustic gradient (TAG) such as that disclosed in U.S. Pat. No. 9,143,674, which is hereby incorporated herein by reference in its entirety.)
determining a condition of the identified component using the recorded image. (see FIG. 3-8 where the component has slope imperfections that are compared to the neural network classification images to infer a problem with the component being photographed; The video tool portion 143 also includes Z-height measurement tools portion 143 z, which provides various operations and features related to Z-height measurement operations. In one implementation, the Z-height measurement tools portion 143 z may include Z-height tools 143 zt. The Z-height tools 143 zt may include an autofocus tool 143 af, and a multipoint autofocus tool 143 maf, for example. The Z-height tools 143 zt may govern certain aspects of image stack acquisition and related structured light pattern generation operations in conjunction with the Z-height tools that are configured in a mode that determines best focus heights and/or Z-height measurements. In general, the Z-height measurement tools portion 143 z may perform at least some operations similarly to known Z-height measurement tools, for example, performing operations in a learn mode and/or run mode or other mode, for generating all or part of a focus curve, and finding its peak as a best focus position. For example, certain known operations for Z-height measurement tools are described in U.S. Pat. No. 10,520,301, which is hereby incorporated herein by reference in its entirety.
The defect detection portion 140 dp performs various defect detection operations, as will be described in more detail below. In various implementations, the defect detection portion 140 dp utilizes models that require training data. In various exemplary implementations, the models may be supervised (e.g., artificial intelligence (AI), etc.) models. The defect detection portion 140 dp processes image data corresponding to user labeled images of defects to train a classification model, which in various implementations may be an AI classification model. The number of images that are required to achieve accurate model predictions on test images can vary greatly (e.g., from about 25 to 300), depending upon factors such as defect and background complexity, variation, and visual differentiation. The defect detection portion 140 dp analyzes a set of defect images provided and labeled by a user, and provides the user with an indication as to whether additional defect images should be provided for the training. As an example, in various implementations the user may be provided with a recommendation (e.g., a yes or no recommendation) as to whether or not more images will improve the performance. As another example, an estimate may be provided of how many defect images are needed in order to train the classification model such that it can perform high accuracy classification. As disclosed herein, such determinations and recommendations are unique to the images provided by the user, characteristics of defects, and the particular model being used, rather than being a generic recommendation of a predetermined number of images (e.g., 100 images). Such guidance enables the user to provide a sufficient quantity of defect images to achieve a model with excellent accuracy classification ability, without taxing the user to provide additional images when little model accuracy improvement is expected from such additional images.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Watson is silent but BURCH teaches “...2. The method of claim 1, wherein the determining of a condition of the identified component comprises using operational data associated with the component, the operational data associated with the component comprising data collected during a vehicle testing phase of the remotely operated vehicle or data collected while the remotely operated vehicle operates”. (see paragraph 68-69 where the drone can inspect the items for a defect but also as they are loaded into the plane to determine if the items are falling down or not loading correcting in addition to capturing the images ]
In some embodiments, sensor array 230 may also include an image sensor as another type of sensing element. Such an image sensor, as part of sensor array 230, may capture images of the items being shipped as the internal monitor drone 125 transits the airborne monitoring path within the internal shipment storage area 120. In other words, the images captured by such an image sensor are from different airborne locations within the shipment storage 110 as the internal monitor drone 125 transits the airborne monitoring path within the interior shipment storage area 120. For example, as internal monitor drone 125 enters an active monitoring state and moves from a secured position on internal docking station 130 to above shipping item 140 b, an image sensor from sensor array 230 may capture images (e.g., still pictures or video; visual images; and/or thermal images) that may be used as sensory information for detecting a condition of the shipping item 140 b (e.g., a broken package for shipping item 140 b, a leak coming from shipping item 140 b, etc.). Exemplary image sensor may be implemented with a type of camera that captures images, thermal images, video images, or other types of filtered or enhanced images that reflect the contents of the internal shipment storage area 120 and provide information about the status of the shipping items within that area 120. Exemplary image sensor may also read and provide imagery or other information that identifies an asset number on an item maintained within the internal shipment storage area 120 (which may eliminate the need for barcode scanning).
[0069]
In further embodiments, sensor array 230 may also include a depth sensor as a further type of sensing element that may make up the array. This depth sensor may be a depth-sensing camera or stereo camera that can interactively capture or map a configuration of the interior shipment storage area 120 of the shipment storage 110 as the internal monitor drone 125 transits the airborne monitoring path within the interior shipment storage area 120. This configuration of the interior shipment storage area represents a multi-dimensional mapping of at least the items being shipped within the interior shipment storage area 120 of the shipment storage 110 (i.e., shipping items 140 a-145 e as shown in FIGS. 1B and 1C). As will be discussed in more detail below, comparisons of such mapped configurations of the interior shipment storage area 120 over time allow for detection of a movement condition for one or more items in the area 120 as monitored from the aerial positions by the internal monitor drone 125. This may be especially helpful during transit as aircraft 100 is airborne and emerges from rough weather conditions where turbulence may have been experienced, and robust monitoring with aerially coordinated depth sensing can check for loose shipping items and help avoid dangerous in-flight cargo scenarios. Additional embodiments may use an ultrasonic transducer as a type of depth sensor that uses sound ways to map surfaces or to help validate data received by a depth sensor camera.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Watson discloses “...3. The method of claim 1, wherein the determining of a condition of the identified component comprises comparing a freshly recorded image of the component with a previously recorded image of said component”. (see FIg. 3-4 where successive images are taken as fresh images and compared with the ai neural network to provide what the images should look like that there are defects in the images and therefore, the part is defective; )Another factor may be in regard to how much the image background varies in visual characteristics. For example, it can be seen that the image backgrounds in FIGS. 3A-3F and 4A-4E each have some amount of variance in certain visual characteristics (e.g., in accordance with a variety of different types of machining marks formed on the plate surfaces which appear as a hashed texture that normally varies across the surface of the plates). In regard to such an example image data set as may include the images of FIGS. 3A-3F and 4A-4E, in general more defect images may be required for the training to obtain a certain level of accuracy than would be required for an image data set including images in which the backgrounds had less variance in visual characteristics (e.g., assuming other factors were approximately equal, etc.) As a specific example, in the images of FIGS. 5A-5D, the backgrounds are more similar (i.e., the types of machining marks formed on the surfaces of the portions of the plates are similar or nominally the same in each image and the primary difference between the images is the characteristics of the respective scratch defects 502B, 502C and 502D). In regard to such an example image data set as may include the images of FIGS. 5A-5D (and for which other images of the set may include similar backgrounds), in general less defect images may be required for the training to obtain a certain level of accuracy than would be required for an image data set including images in which the backgrounds had more variance in visual characteristics, such as illustrated by the images of FIGS. 3A-3F and 4A-4E (e.g., assuming other factors were approximately equal, etc.)
Another factor may be in regard to the image quality (e.g., as to whether the image quality is consistent and good in regard to focus, lighting, etc. and/or if it may vary for the different images in the image data set). For example, it can be seen that the images in FIGS. 3A-3F, 4A-4E and 5A-5D each appear to have relatively good focus, etc. In regard to such an example image data set as may include such images, in general less defect images may be required for the training to obtain a certain level of accuracy than would be required for an image data set including images in which the images had less image quality (e.g., assuming other factors were approximately equal, etc.) As a specific example, in an image data set with similar images except where all of the images had less focus (e.g., for which it may correspondingly be more difficult to accurately determine the characteristics of the scratch defects, backgrounds etc. and/or to differentiate the scratch defects from the backgrounds, etc.), in general more defect images may be required for the training to obtain a certain level of accuracy (e.g., assuming other factors were approximately equal, etc.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Watson discloses “..4. The method of claim 1, wherein the determining of a condition of the identified component comprises using batch information associated with said component”. (see Fig. 9 where a number of workpieces can all be inspected together and then the workpieces can be determined to be defective and then the entire batch is provided for scrap; FIG. 9 shows a flow diagram of a method 900 for operating a machine vision inspection system similar to that of FIG. 1. The method 900 begins at 910. At 910, a plurality of images of one or more workpieces is acquired. For example, the camera system 260 acquires a plurality of images of workpieces 20. The method 900 then proceeds to 920.
At 920, each of the images of the workpieces acquired at 910 is classified as non-defect or defect. For example, a user visually inspects each image acquired at 910. In various implementations, if an image is deemed to not include a defect, a predetermined field included in a header of a file that includes image data corresponding to the image is set to a predetermined value that indicates the image is a non-defect image (e.g., 0). If an image is deemed to include a defect, the predetermined field included in the header of the file that includes image data corresponding to the image is set to a predetermined value that indicates the image is a defect image (e.g., 1). The method 900 then proceeds to 930.
At 930, for each of the images of the workpieces that is classified as a defect at 920, a plurality of pixels that correspond to a defect is determined. For example, a user visually inspects each image classified as a defect, and then uses a software tool that enables the user to set a color value (e.g., grayscale value) to a predetermined value (e.g., 255) of each pixel in an overlaid transparent image corresponding to each defect. The method 900 then proceeds to 940.
At 940, model training is determined. For example, the defect detection portion divides the images acquired at 910, based on how those images are classified at 920, into a plurality of groups in a manner similar to that described above in connection with FIG. 7. Also, a user may select a particular one of a plurality of models included in the defect detection portion using a graphical user interface. The method 900 then proceeds to 950.
At 950, model training is initiated based on the model training determined at 940. For example, such initiation may be performed automatically, or a user may push a button or select an icon using a graphical user interface, which causes the model training to begin based on the model training determined at 940. The method 900 then ends.
FIG. 10 shows a flow diagram of a method 1000 for operating a machine vision inspection system similar to that of FIG. 1. The method 1000 begins at 1010. At 1010, a first plurality of training images of one or more workpieces acquired with a camera (e.g., camera system 260) is utilized to train a defect detection portion (e.g., defect detection portion 140 dp) to detect defect images that include a workpiece with a defect. The method 1000 then proceeds to 1020.
At 1020, a first plurality of testing images of the workpieces acquired with the camera, and not included in the first set of training images, is utilized to test the defect detection portion as trained using the first set of training images. A first testing image performance metric is determined from the test. For example, the performance metric may be calculated by dividing the number of training images accurately classified as being a defect image by the total number non-defect images in the first plurality of testing images. The method 1000 then proceeds to 1030.
At 1030, the first plurality of training images and a second plurality of training image acquired with the camera are utilized to train the defect detection portion to detect defect images. The method 1000 then proceeds to 1040.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
BURCH teaches “...5. The method of claim 1, wherein the determining of a condition of the identified component is based on environmental conditions associated with the automated storage and retrieval system comprising ambient temperature or ambient pressure. (see paragraph 129-133 where the drone will also include sensors that can provide a signal about the environment to determine if this is a source of the defect FIG. 9 shows a flow diagram of a method 900 for operating a machine vision inspection system similar to that of FIG. 1. The method 900 begins at 910. At 910, a plurality of images of one or more workpieces is acquired. For example, the camera system 260 acquires a plurality of images of workpieces 20. The method 900 then proceeds to 920.
At 920, each of the images of the workpieces acquired at 910 is classified as non-defect or defect. For example, a user visually inspects each image acquired at 910. In various implementations, if an image is deemed to not include a defect, a predetermined field included in a header of a file that includes image data corresponding to the image is set to a predetermined value that indicates the image is a non-defect image (e.g., 0). If an image is deemed to include a defect, the predetermined field included in the header of the file that includes image data corresponding to the image is set to a predetermined value that indicates the image is a defect image (e.g., 1). The method 900 then proceeds to 930.
At 930, for each of the images of the workpieces that is classified as a defect at 920, a plurality of pixels that correspond to a defect is determined. For example, a user visually inspects each image classified as a defect, and then uses a software tool that enables the user to set a color value (e.g., grayscale value) to a predetermined value (e.g., 255) of each pixel in an overlaid transparent image corresponding to each defect. The method 900 then proceeds to 940.
At 940, model training is determined. For example, the defect detection portion divides the images acquired at 910, based on how those images are classified at 920, into a plurality of groups in a manner similar to that described above in connection with FIG. 7. Also, a user may select a particular one of a plurality of models included in the defect detection portion using a graphical user interface. The method 900 then proceeds to 950.
At 950, model training is initiated based on the model training determined at 940. For example, such initiation may be performed automatically, or a user may push a button or select an icon using a graphical user interface, which causes the model training to begin based on the model training determined at 940. The method 900 then ends.
FIG. 10 shows a flow diagram of a method 1000 for operating a machine vision inspection system similar to that of FIG. 1. The method 1000 begins at 1010. At 1010, a first plurality of training images of one or more workpieces acquired with a camera (e.g., camera system 260) is utilized to train a defect detection portion (e.g., defect detection portion 140 dp) to detect defect images that include a workpiece with a defect. The method 1000 then proceeds to 1020.
At 1020, a first plurality of testing images of the workpieces acquired with the camera, and not included in the first set of training images, is utilized to test the defect detection portion as trained using the first set of training images. A first testing image performance metric is determined from the test. For example, the performance metric may be calculated by dividing the number of training images accurately classified as being a defect image by the total number non-defect images in the first plurality of testing images. The method 1000 then proceeds to 1030.
At 1030, the first plurality of training images and a second plurality of training image acquired with the camera are utilized to train the defect detection portion to detect defect images. The method 1000 then proceeds to 1040.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
BURSCH teaches “...6. The method of claim 1, wherein identifying the component based on the recorded image comprises reading ID-data associated with the component.” (see paragraph 5 where the components can include an RFID tag and 57-60 where the drone can scan an RFID tag for the packages to identify what is damaged; )”/
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Claim 7 is rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. That was filed in 2020 and in view of United States Patent Application Pub. No.: US20210073727A1 to Burch and in view of Chinese Patent Pub. No.: CN113908373B to Ma filed in 2015.
Watson is silent but Ma teaches “..7. The method of claim 1, wherein the determined condition of the identified component is remaining useful life (RUL)”. (see abstract and claims 1-2)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of MA with a reasonable expectation of success since that MA teaches that the part can fail and also how much longer the part can be provided in its current use before a replacement. Then a user can delay the replacement and save costs.
Claim 8 is rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. That was filed in 2020 and in view of United States Patent Application Pub. No.: US20210073727A1 to Burch and in view of United States Patent Application Pub. No.: US20210303378A1 to Sethi.
Sethi teaches “...8. The method of claim 1, wherein the determined condition of the component is probability of component failure within a given time window” (see abstract and claims 1-11 where the part is determined to fail presently and a warranty may be claimed due to the fact that this part is about to fail based on the telemetry data).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of SETHI with a reasonable expectation of success since SETHI teaches that a telemetry data and other parameters can be input into the algorithm. Then the part can be predicted to be in a failure state shortly and a warranty claim can be made now so as to anticipate the failure while the vehicle is still working. This can prevent any downtime.
Claim 9 is rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. That was filed in 2020 and in view of United States Patent Application Pub. No.: US20210073727A1 to Burch and in view of United States Patent Application Pub. No.: US20170344909A1 to KUROKAWA.
KURKOWA teaches “..9. The method of claim 1, wherein a time for next vehicle maintenance is set based on the determined condition of the vehicle component. (see paragraph 12-30 where a condition can provide an update on the end of life of the component calculation including how the component is being used)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of KUROKAWA with a reasonable expectation of success since KUROKAWA can include a parameter as to the use of the component to update the end of life parameter. This can provide an increased accuracy and if lightly used this can prolong the life.
Claims 10-15 are rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. that was filed in 2020 and in view of United States Patent Application Pub. No.: US20210073727A1 to Burch
Watson discloses “...10. The method of claim 1, wherein, if the determined condition of the identified component is not satisfactory, an image of a selected section of the identified component is recorded”. (see FIG. 3-8 where the surface of the workpiece may be so bad in the image that it is provided for scrap and discarded and the image can be provided to the neural network for recording;lso, some defects might warrant further metrology or inspection to determine additional defect parameters. For example, a 2D image could enable a likely defect to be quickly recognized and enable an XY position of the defect and an approximate XY area of the defect to be quickly ascertained. If the 3D nature of a potential defect is important, the defect detection portion 140 dp may cause additional processing (e.g., metrology operations) to be performed to determine whether the potential defect is an actual defect. For example, if a scratch in a surface of a workpiece must be deeper than a particular threshold value to be considered a defect, the defect detection portion 140 dp could cause a more time consuming 3D point cloud of the affected region to be acquired (e.g., utilizing a z-height measurement tools portion 143 z), to learn if the depth of the scratch is sufficient to cause the part to be rejected. In various implementations, different actions may be performed (e.g., as may be programmed to automatically occur) as a result of an initial defect classification, such as (1) continue with a standard metrology process, (2) stop or pause the defect detection process and perform a metrology process that includes more informative measures of the potential defect (e.g., 3D, different lighting, touch probe measurements such as surface roughness, etc.), (3) send the workpiece to scrap (e.g., discard or recycle workpiece), (4) send the workpiece for additional human inspection, (5) provide feedback to a production line that indicates something may be wrong with the machinery or process, etc. )
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
BURCH teaches “...11. The method of claim 10, wherein said section of the component is selected based on at least one of operational data associated with the component comprising data collected during a vehicle testing phase or data collected while the vehicle operates, batch information associated with the component, or environmental conditions associated with the automated storage and retrieval system comprising ambient temperature or ambient pressure”. (see paragraph 129-133 where the drone will also include sensors that can provide a signal about the environment to determine if this is a source of the defect FIG. 9 shows a flow diagram of a method 900 for operating a machine vision inspection system similar to that of FIG. 1. The method 900 begins at 910. At 910, a plurality of images of one or more workpieces is acquired. For example, the camera system 260 acquires a plurality of images of workpieces 20. The method 900 then proceeds to 920.
At 920, each of the images of the workpieces acquired at 910 is classified as non-defect or defect. For example, a user visually inspects each image acquired at 910. In various implementations, if an image is deemed to not include a defect, a predetermined field included in a header of a file that includes image data corresponding to the image is set to a predetermined value that indicates the image is a non-defect image (e.g., 0). If an image is deemed to include a defect, the predetermined field included in the header of the file that includes image data corresponding to the image is set to a predetermined value that indicates the image is a defect image (e.g., 1). The method 900 then proceeds to 930.
At 930, for each of the images of the workpieces that is classified as a defect at 920, a plurality of pixels that correspond to a defect is determined. For example, a user visually inspects each image classified as a defect, and then uses a software tool that enables the user to set a color value (e.g., grayscale value) to a predetermined value (e.g., 255) of each pixel in an overlaid transparent image corresponding to each defect. The method 900 then proceeds to 940.
At 940, model training is determined. For example, the defect detection portion divides the images acquired at 910, based on how those images are classified at 920, into a plurality of groups in a manner similar to that described above in connection with FIG. 7. Also, a user may select a particular one of a plurality of models included in the defect detection portion using a graphical user interface. The method 900 then proceeds to 950.
At 950, model training is initiated based on the model training determined at 940. For example, such initiation may be performed automatically, or a user may push a button or select an icon using a graphical user interface, which causes the model training to begin based on the model training determined at 940. The method 900 then ends.
FIG. 10 shows a flow diagram of a method 1000 for operating a machine vision inspection system similar to that of FIG. 1. The method 1000 begins at 1010. At 1010, a first plurality of training images of one or more workpieces acquired with a camera (e.g., camera system 260) is utilized to train a defect detection portion (e.g., defect detection portion 140 dp) to detect defect images that include a workpiece with a defect. The method 1000 then proceeds to 1020.
At 1020, a first plurality of testing images of the workpieces acquired with the camera, and not included in the first set of training images, is utilized to test the defect detection portion as trained using the first set of training images. A first testing image performance metric is determined from the test. For example, the performance metric may be calculated by dividing the number of training images accurately classified as being a defect image by the total number non-defect images in the first plurality of testing images. The method 1000 then proceeds to 1030.
At 1030, the first plurality of training images and a second plurality of training image acquired with the camera are utilized to train the defect detection portion to detect defect images. The method 1000 then proceeds to 1040.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Watson discloses “...12. The method of claim 1, wherein the recorded image of the component is recorded by a camera”. (FIG. 9 shows a flow diagram of a method 900 for operating a machine vision inspection system similar to that of FIG. 1. The method 900 begins at 910. At 910, a plurality of images of one or more workpieces is acquired. For example, the camera system 260 acquires a plurality of images of workpieces 20. The method 900 then proceeds to 920.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Watson discloses “...13. The method of claim 12, wherein information regarding the identified component is presented on a display”. (See FIg. 12 where at 1240, whether an action is to be performed is determined based on dimensions and/or characteristics determined at 1230. For example, if a dimension of the defect 502D is determined to be above a threshold, an action may be performed (e.g., an alert may be provided, the workpiece may be sent to scrap, etc.) In one specific example implementation, the defect detection portion may determine whether the maximum depth of the defect 502D is greater than or equal a predetermined threshold value. If the defect detection portion determines that the maximum depth of the defect 502D is greater than or equal to the predetermined threshold value, the defect detection portion may determine that a message is to be generated and displayed (e.g., indicating information about the defect and/or that the workpiece should be send to scrap, etc.) If a dimension or other characteristic (e.g., of the defect 502D) is not above a threshold, the defect detection portion may determine that no action is to be performed. The method 1000 then ends.)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Watson discloses “...14. The method of claim 1, wherein the recorded image of the component is recorded by at least one fixed-position camera”. (See FIg. 2 where in various exemplary implementations, the optical assembly portion 205 may further include a turret lens assembly 223 having lenses 226 and 228. As an alternative to the turret lens assembly, in various exemplary implementations a fixed or manually interchangeable magnification-altering lens, or a zoom lens configuration, or the like, may be included. In various exemplary implementations, the interchangeable objective lens 250 may be selected from a set of fixed magnification objective lenses that are included as part of the variable magnification lens portion (e.g., a set of objective lenses corresponding to magnifications such as 0.5×, 1×, 2× or 2.5×, 5×, 10×, 20× or 25×, 50×, 100×, etc.).)It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
BURCH teaches “..15. The method of claim 14, wherein the remotely operated vehicle is arranged to be rotated prior to the recording of the image of the component”. (see 63-67 where the drone can be provided with any flight path to inspect the larger aircraft or the larger components of the device for determining if the parts are defective and or damaged)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Claim 16 is rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. That was filed in 2020 and in view of United States Patent Application Pub. No.: US20210073727A1 to Burch and in view of United States Patent No.: US8406923B2 to TAKAHASHI.
TAKAHAYSHI teaches “...16. The method of claim 1, wherein the recorded image of the component is recorded by at least one camera of a movable robotic arm”. (see claim 1-5 and the abstract)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of TAKAHAYSHI with a reasonable expectation of success since TAKAHAYSHI that a drone can include an arm and a mounted camera on the arm for improved positioning of the camera a desired.
Claims 17-19 are rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. That was filed in 2020 and in view of United States Patent Application Pub. No.: US20210073727A1 to Burch,
BURCH teaches “...17. The method of claim 1, wherein selecting the component of the remotely operated vehicle is preceded by identifying the remotely operated vehicle.(see paragraph 142-150 where the inspection drone can inspect the larger vehicle including the conveyor belt and the landing gear)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
BURCH teaches “...18. The method of claim 17, wherein the identifying of the remotely operated vehicle comprises recording an image of the remotely operated vehicle. (see paragraph 142-150 where the inspection drone can inspect the larger vehicle including the conveyor belt and the landing gear)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
BURCH teaches “...19. The method of claim 1, wherein said component is a wheel of the vehicle, a lifting device for vertical transportation of goods holders, a gripper element, a gearbox, a motor belt, or a goods holder contact sensor”. (see paragraph 142-150 where the inspection drone can inspect the larger vehicle including the conveyor belt and the landing gear)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Claim 21 is rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. and in view of United States Patent No.: US12281989B2 to Gupta.
Gupta teaches “..21. The system of claim 20, wherein said inspection device is a handheld device comprising a display” (see claims 1-8 and the abstract where the handheld device can include an IR device for inspection of a part).
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of GUPTA with a reasonable expectation of success since GUTPA teaches that a handheld device can include an IR camera. The IR camera can provide an inspection of the part. The IR camera can also be provided with an indication of an inspection of the part and if the part is faulty.
Claims 22-24 and 26-27 are rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al.
Watson discloses “...22. The system of claim 20, wherein said system comprises a booth for accommodating a remotely operated vehicle, said booth being provided with said inspection device comprising the camera. (see paragraph 150 where the drone can provide a designated inspection area for inspecting the parts or boxes or a second drone)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Watson discloses “...23. The system of claim 22, wherein said camera is at least one fixed-position camera”.(See FIg. 2 where in various exemplary implementations, the optical assembly portion 205 may further include a turret lens assembly 223 having lenses 226 and 228. As an alternative to the turret lens assembly, in various exemplary implementations a fixed or manually interchangeable magnification-altering lens, or a zoom lens configuration, or the like, may be included. In various exemplary implementations, the interchangeable objective lens 250 may be selected from a set of fixed magnification objective lenses that are included as part of the variable magnification lens portion (e.g., a set of objective lenses corresponding to magnifications such as 0.5×, 1×, 2× or 2.5×, 5×, 10×, 20× or 25×, 50×, 100×, etc.).)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Burch teaches “...24. The system of claim 22, wherein the remotely operated vehicle is rotatable when positioned in said booth. (see 63-67 where the drone can be provided with any flight path to inspect the larger aircraft or the larger components of the device for determining if the parts are defective and or damaged)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Claim 25 is rejected under 35 U.S.C. sec 103 as being unpatentable as obvious in view of United States Patent Application Pub No. 11150200 B1 to Watson et al. and in view of TAKAHASHI.
Takahashi discloses “...25. The system of claim 22, wherein said camera is at least one camera provided on a movable robotic arm.(see claim 1-5 and the abstract)
See motivation statement above.
BURCH teaches “..26. The system of claim 20, wherein said component is a wheel of the vehicle, a lifting device for vertical transportation of goods holders, a gripper element, a gearbox, a motor belt or a goods holder contact sensor. (see paragraph 142-150 where the inspection drone can inspect the larger vehicle including the conveyor belt and the landing gear)
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
BURCH discloses “...27. An automated storage and retrieval system comprising a framework structure that comprises a plurality of storage columns for storing goods holders, wherein said automated storage and retrieval system comprises a system for determining a condition of a component of a remotely operated vehicle operating on a rail system by executing steps of the method in accordance with claim 1. “ (see claims 68-75 where the drone can provide a flight plan to provide an inspection of a component including a second drone or flight device: and see FIG. 9-11 show At 1050, based at least in part on the first and second testing image performance metrics, a recommended number of defect images for training the defect detection portion is determined. For example, the defect detection portion fits a training image performance versus number of defect images used for model training results with an appropriate curve, fits a testing image performance versus number of defect images used for model training results with an appropriate curve, and utilizes a convergence algorithm to find a value of the number of defect images used to train the model for which the testing fitted curve and the training fitted curve converge toward each other, in a manner similar to that described above in connection with FIG. 7. The method 1000 then proceeds to 1060.
At 1060, after at least the recommended number of defect images have been used to train the defect detection portion, the camera is utilized to acquire new images of workpieces during a run mode, and the defect detection portion is utilized to analyze the new images to determine defect images that include workpieces with defects (e.g., including classifying the new images a defect images or non-defect images). The method 1000 then ends.
FIG. 11 shows a flow diagram of a method 1100 for operating a machine vision inspection system similar to that of FIG. 1. For example, in a manner similar to that described above in connection with FIG. 7, a defect detection portion is trained using a plurality of groups of training images, wherein each group of training images includes a predetermined number of defect images. More particularly, the defect detection portion is trained during a series of training cycles. During the first training cycle, the defect detection portion is trained with a first group of training images. During the second training cycle, the defect detection portion is trained with the first group of training images along with a second group of training images. During the third training cycle, the defect detection portion is trained with the first and second groups of training images along with a third group of training images. During the fourth training cycle, the defect detection portion is trained with the first, second, and third groups of training images along with a fourth group of training images. After each training cycle, the accuracy of the defect detection portion is determined using two sets of images, including the set of training images used to train the defect detection portion in the previous training cycle, and a set of testing images not used to train the defect detection portion in the previous cycle. )
It would have been obvious for one of ordinary skill in the art before the effective filing date of the present disclosure to combine the disclosure of WATSON with the teachings of BURCH with a reasonable expectation of success since that a small drone can include a flight plan to inspect a larger container vehicle like a fedex plane or truck to inspect all of the items within the plane by taking images and then also inspect the plane and larger vehicle itself. This can provide an improved productivity and a human does not need to do a walkthrough which can be automated to increase profit for the provider. See abstract and claims 1-10.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEAN PAUL CASS whose telephone number is (571)270-1934. The examiner can normally be reached Monday to Friday 7 am to 7 pm; Saturday 10 am to 12 noon.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott A. Browne can be reached at 571-270-0151. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEAN PAUL CASS/Primary Examiner, Art Unit 3666
Prosecution Insights