Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The abstract of the disclosure is objected to because there is punctuation missing after the sentence “The inspection setting section receives setting of a window for a learning image on which the object appears”. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: ‘image capturing section’, ‘inspection execution section’, and ‘inspection setting section’ in claim 1.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Do (US20210390677A1) in view of Milne (WO2021225876A1) and further in view of Kumar (A Real-Time Object Counting and Collecting Device for Industrial Automation Process Using Machine Vision).
Regarding claim 1, Do teaches an image inspection device comprising: an image capturing section that continuously captures a capturing field of view to generate a plurality of frame images aligned in time series (“The received data can be derived from a video feed (i.e., a series of images, etc.)" [0010], and "the user can specify a “continuous” trigger mode, in which all frames in which the object appears can be captured and made available” [0115]); an inspection execution section that executes inspection processing of an object appearing in the plurality of frame images to output an inspection result (“the current subject matter simplifies manufacturing, procurement and configuration of the hardware and software components required to install and obtain value from a camera-based quality assurance inspection system" [0051], and "complementary information can take various forms including, for example, various quality assurance metrics such as dimensions, color, and the like" [0089], further noting that “in the vision processing pipeline the images are analyzed by inspection tools which utilize computer vision or machine learning algorithms to determine quality assurance metrics with respect to the images” [0084]); and an inspection setting section that performs setting of the inspection execution section (“the inspection routine configuration 235 can be such that any analyses are constrained to a region of the frame (e.g., a region of interest, or ROI). The region of interest can be specified by the user in a user interface. The method by which the user sets the region of interest may be drawing a rectangle or other polygon on a reference image” [0116], noting that “the image analysis inspection tool 240.sub.0 can perform an alignment in which the raw image 215 is compared to a reference image… ); wherein the inspection setting section receives setting of a window (“ The salient object detector can use a convolutional neural network (CNN) that produces a mask that indicates, via a probability on a per pixel basis, where the foreground objects are likely to be. The CNN can be trained on a dataset of thousands of objects with per pixel labels indicating where in the images the salient objects are located” [0112], noting further that “A vision processing pipeline system 220 receives the raw image 215 and selectively (and optionally) crops 225 aspects of the image based on areas of interest (AOIs) to result in two or more AOIs 230.sub.1, 230.sub.2" [0087], see Figs. 6 and 7); and the inspection execution section detects target regions from the plurality of frame images (noting the salient object detector as disclosed above at [0112], “The detecting can include generating, using at least one machine learning model, a mask that indicates, via a probability on a per pixel basis, a likely location of an object within each image. The at least one machine learning model can take various forms including a convolutional neural network (e.g., a fully convolutional network, etc.) that is trained using a dataset with per pixel labels indicating locations of salient objects within images” [0033]); acquires first positional information of the target region detected in a first frame image and second positional information of the target region detected in a second frame image (“The algorithm can use a CNN that accepts as input two image frames that form a sequence and produces an array indicating the motion between the two frames” [0113]); performs matching processing of the target region between the first frame image and the second frame image based on the first positional information, the second positional information, and an overall moving direction of the object (“The pixels in the output array correspond to a length 2 vector that indicates where the pixel in the first frame ended up moving to in the second frame” [0113]). Do does not teach training ‘learning images’ based on the setting of a window.
However, Milne teaches training ‘learning images’ based on the setting of a window (“if a particular neural network is intended to detect defects in a relatively small/constrained region of the container, the training images for that model may be generated by automatically cropping the original container images to exclude at least some areas outside of the region of interest” [0005]). Examiner further notes that the user may also manually set the window, as per [00149], “With object detection, an AVI neural network is shown what area to focus on (e.g., via a bounding box or other boundary drawn manually using a labeling tool, or via an area that is manually “painted” using a labeling tool), and the neural network returns/generates a bounding box”.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the camera-based quality assurance inspection device disclosed by Do to include the pre-training region of interest cropping method for object inspection as disclosed by Milne, the proposed motivation being that training models using only the regions of interest of images, especially when high volumes of images need to be used, reduces the processing time and memory space needed when images with fewer pixels can be used to achieve an identical result.
Both Do and Milne fall silent regarding counting the object based on a result of the matching processing. However, Kumar teaches counting the object based on a result of the matching processing (“we tracked the centroid of an object in this region and determined the displacement in their location in consecutive frames” [page 13055], and “the objects are counted while crossing the boundary, counted remains on one side and uncounted objects remain on the other side of the boundary” [page 13053]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the camera-based quality assurance inspection device disclosed by Do and in view of Milne to include object counting capabilities as disclosed by Kumar, the proposed motivation being that a quality inspection device that also counts the inspected objects further automates a process that would otherwise require lots of time, precision, and energy by human employees in quality assurance applications, especially of small, mass produced components that require precision and accuracy for proper functioning.
Regarding claim 2, Do in view of Milne and Kumar teaches the image inspection device according to claim 1. Do further teaches wherein the inspection setting section stores an image characteristic of the object based on the setting of the window (“Machine learning models, in particular, can often perform poorly on “out of sample” images, where “out of sample” images are images on which the model has not been configured or trained. It can be useful to come up with a score, hereafter referred to as the “visual similarity score”, which can be a floating-point or integer number which represents how similar an image 215 is to the set of reference and/or training image or images on which the image analysis inspection tool was configured” [0100], noting that, “Optimally represented can take various forms including a most centered representation of the object across all images including the object…Optimally represented can include a representation of the object having image properties closest to a set of reference images on which a quality assurance inspection tool was configured or trained. The image properties can be based on various aspects including one or more of sharpness, blurriness, color space or visual similarity score” [0030]. Examiner notes that ‘a most centered representation of the object’ requires resizing/cropping/’setting of the window’) and the inspection execution section detects the target regions from the plurality of frame images based on the image characteristic of the object (“The detector may also use a visual similarity score for the image, as discussed above, to determine which image of the item most closely matches the user configuration” [0112]).
Regarding claim 3, Do in view of Milne and Kumar teaches the image inspection device according to claim 2. Do further teaches wherein the inspection setting section extracts and stores a first feature amount corresponding to the image characteristic of the object from the learning image by inputting the learning image to a machine learning model (“The visual similarity score…may be a mathematical algorithm which analyzes the average color value of the pixels of the image 215 and compares this to the average pixel value of the training and/or reference image or images to determine the score…the visual similarity score may be used as input to the inspection tool” [0100], noting at [0085], “machine learning algorithm, referred to herein as the inspection tool”) and the inspection execution section detects the target region in which a second feature amount closest to the first feature amount is obtained from the frame image by inputting the frame image to the machine learning model (“machine learning algorithms utilized in such a system typically perform complex mathematical processes in order to determine the quality assurance metrics associated with a given image. These mathematical processes typically involve receiving a two-dimensional or three-dimensional image as an input, performing mathematical operations on the image or regions of the image, and then producing a series of quality assurance metrics” [0084], noting from [0085] that the ‘second feature amount’ is the average color value of the pixels of the image, which is distinct from the ‘first feature amount’ which was the average pixel value of the training image).
Regarding claim 4, Do in view of Milne and Kumar teaches the image inspection device according to claim 1. Do further teaches wherein the inspection setting section receives setting of the overall moving direction (“The user can also use a graphical user interface (GUI) in order to configure the expected direction in which items are expected to flow through the camera's field of view (FOV). The user interface (UI) for configuring this, can include an arrow which the user draws on a reference image to indicate the expected flow direction of objects” [0116]).
Regarding claim 5, Do in view of Milne and Kumar teaches the image inspection device according to claim 1. Kumar further teaches wherein a detection range of the object is divided into a pre-passage region and a post- passage region (Examiner notes that per Fig. 6 provided below, the ‘pre-passage region’ is the location to the left of the yellow (rightmost) line (see frame b) and that the ‘post-passage region’ is the location to the right of the yellow line (see frames c-e). Examiner notes further that the blue (leftmost) line is present simply as a point for the computer to recognize that it no longer needs to store centroid information of the object in frame, but the count information is still stored. Examiner also notes that the object is moving from right to left in the camera FOV), and the inspection execution section counts only the target region determined to be positioned in the pre-passage region in the first frame image and determined to be positioned in the post-passage region in the second frame image, based on the result of the matching processing (“we tracked the centroid of an object in this region and determined the displacement in their location in consecutive frames. As all the objects are moving on the same conveyor belt, they have identical speeds and will travel identical and predictable distances in consecutive frames” [page 13055], additionally note that per frame c of Fog. 6 below, “Fig. 6(c) describes the instance when the centroid of the object detected has crossed the right vertical line/boundary, and thus, it has been counted as the first object. Hence, the counter is increased by 1, and the centroid is drawn as a point and boundary is marked across the object as shown in the figure” [page 13056] ).
PNG
media_image1.png
132
510
media_image1.png
Greyscale
Regarding claim 6, Do in view of Milne and Kumar teaches the image inspection device according to claim 5. Kumar further teaches wherein the inspection setting section receives setting of a passage line that divides the pre-passage region and the post-passage region (“the region that lies between two lines depends on the size of the object and must be set before starting the program for the final count” [page 13054]. Examiner notes that setting the region between the two lines of interest (see Fig. 6 above) inherently requires configuring an ideal location for the lines themselves).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the camera-based quality assurance inspection device disclosed by Do and in view of Milne and Kumar further to use the passage line separating the pre-passage and post-passage regions as disclosed by Kumar as input into the inspection setting section of the proposed device, the suggested motivation being that allowing for the user to set an ideal location of the passage line for a specific object detection/counting application and input that into the inspection setting section allows for device use in a wider range of counting, detection, and inspection needs.
Regarding claim 8, Do in view of Milne and Kumar teaches the image inspection device according to claim 1. Milne further teaches wherein the inspection execution section detects an abnormal state by comparing detection results of the target regions (“To accomplish this, module 136 examines a given heatmap (e.g., heatmap 1600) generated by a neural network that is trained to detect a particular class of defects, and determines whether the portions of the image that were most important to a classification made by that neural network are the portions that should have been relied upon to make the inference” [00136], noting that determining which portions ‘should have been relied upon’ is a comparison of actual vs. possible detection results) in the plurality of frame images (Examiner notes that the ‘plurality of frame images’ is disclosed in Do) and outputs a warning (“ the importance of that portion to the inference/classification that the neural network makes for that image (e.g., “good” or “defect” [00131]. Examiner notes that under broadest reasonable interpretation, a “defect” label is considered a ‘warning’, and further, regarding ‘output’ that “The GUI (or another GUI generated by another program) may also display each captured frame/image to the user, and include user interactive controls for manipulating the image (e.g., zoom, pan, etc.) and for manually labeling the image (e.g., “defect observed” or “no defect” for image classification, or drawing boundaries within, or pixel-wise labeling, portions of images for object detection)” [0064]. Examiner notes that in addition to the above disclosure, it is understood that the images that are output to the user contain labels, which warn of a defect for the express purpose of the user reviewing candidate training samples 140 so as to ensure accurately labeled supervised training samples).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the camera-based quality assurance inspection device disclosed by Do and in view of Milne and Kumar to include a specific case for detection of ‘abnormalities’ in image frames as disclosed by Milne, the proposed motivation being that implementing quality assurance analysis on frames containing defect would make for stronger inspection result outcomes.
Regarding claim 9, Do in view of Milne and Kumar teaches the image inspection device according to claim 8. Milne further teaches wherein the inspection setting section receives, as a learning recommendation image, the frame image in a section in which the abnormal state is detected, among the plurality of frame images (“Pooling defect images of different categories in order to balance out numerous good images can result in a robust image library that encapsulates a much broader range of variability and fine detail. Moreover, pooling defects can substantially increase the variability in the “defect” image library, because the different containers from which the defect images were sourced will be from a broader range of sources.” [0080]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the camera-based quality assurance inspection device disclosed by Do and in view of Milne and Kumar to provide recommendation images for further model training as disclosed by Milne, the proposed motivation being that doing so would increase the accuracy and robustness of the proposed model.
Regarding claim 10, Do in view of Milne and Kumar teaches the image inspection device according to claim 1. Milne further teaches wherein the inspection setting section selects the learning image for additional learning from an inspection result screen and receives the setting of the window (“if it is determined at stage 1210 that discriminator 1204 incorrectly classified the input image, then generator 1202 successfully fooled discriminator 1204. In this case, feedback is instead provided to the neural network of discriminator 1204, to further train its neural network” [00114], further noting that “Module 136 may keep a count of such instances for each of the AVI neural networks, for example, and possibly compute one or more metrics indicative of how often each of neural networks makes a classification for the wrong reason” [00138]. Examiner notes that ‘further diversifying the images in image library, in light of [00100], includes “technique 1000 that may be used to check whether an assembled library has sufficient diversity. Technique 1000 may be implemented by software (e.g., module 134) of computer system 104, for example. In technique 1000, at an initial stage 1002, computer system 104 collects (e.g., requests and receives) images from a training library (e.g., image library 140) for analysis. The images may be only non-defect images, only defect images, only images pertaining to a specific defect class, or some mix thereof, for example”) the inspection result screen displaying the frame image (“Computer system 104 may then display such counts and/or metrics to a user, who can determine whether a particular AVI neural network should be further trained (e.g., by further diversifying the images in image library 140, etc.)” [00138]).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Do in view of Milne and Kumar further in view of Chan (US20150271399A1).
Regarding claim 7, Do in view of Milne and Kumar teaches the image inspection device according to claim 5. They all fall silent regarding wherein the inspection setting section receives setting for allowing disappearance of the target region in the post-passage region, and the inspection execution section counts the target region even though the target region disappears in the post-passage region in a case where the inspection setting section receives the setting for allowing disappearance of the target region.
However, Chan teaches wherein the inspection setting section receives setting for allowing disappearance of the target region in the post-passage region (“The photo capturing area is set as that all articles passing through the photo capturing area are at least captured once; the photo capturing area is on a path of the branch article streams after the branch article streams leave the end of the conveyor” [0027]. Examiner notes that per Applicant’s specification at [0091], the disclosed ‘disappearance’ refers to the object dropping off the end of the conveyor belt and into a container below.), and the inspection execution section counts the target region even though the target region disappears in the post-passage region in a case where the inspection setting section receives the setting for allowing disappearance of the target region (“A length of the photo capturing area is set as a distance that an article at a starting line of the photo capturing area moves in a photo capturing period, so that the article is entirely captured only once when the article passes through the photo capturing area or partially captured once in two consecutively captured photos respectively when entering and leaving the photo capturing area, and in each captured photo, when an entire article image appears, or a partial article image intersecting a termination line of the photo capturing area appears, increasing the counting by one” [0029]. Examiner notes that in this embodiment, the target region of the objects is only partially captured in cases when they are leaving the capturing area, as the rest of the object has ‘disappeared’ in the post-passage region by leaving the image frame, concluding that the target objects/image regions disappear once they reach the post-passage region).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the camera-based quality assurance inspection device disclosed by Do and in view of Milne and Kumar to include an option for counting that only requires a single captured frame of the object as taught by Chan, the proposed motivation being to allow for the automation of the packaging process as well, since in order for the packaging process to effectively be automated, the counting must occur right before the objects enter the packages for maximal accuracy.
Additional References
Prior art made of record and not relied upon that is considered pertinent to
applicant’s disclosure:
Additionally cited references (see attached PTO-892) otherwise not relied upon
above have been made of record in view of the manner in which they evidence the
general state of the art. Examiner notes that Tam (Real-Time Object Counting on Industrial Conveyor Belts Using YOLOv11 and Zone-Based Algorithm) discloses a very similar counting method as disclosed in at least claims 1 and 5-7 of Applicant’s disclosure but does not qualify as prior art due to publishing date. Regardless, it is included for Applicant’s reference and for reference to current state of the art. Any/all other cited references may similarly/alternatively serve to anticipate at least the independent claims as recited, and/or provide examples of current art to Applicant as discovered and noted by Examiner during search.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA T RAMETTA whose telephone number is (571)272-0451. The examiner can normally be reached Monday- Friday, 8 a.m. 5 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JULIA T RAMETTA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
