DETAILED ACTION
This action is in response to the application filed on January 11th, 2024. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1-14 and 19 are objected to because of the following informalities:
Claim 1, line 16: “the backside of the plurality of while the plurality of images are being captured” containers a minor typographical error, this should read “the backside of the plurality of containers while the plurality of images are being captured”.
Claim 7, line 3: “the at least one image” lacks antecedent basis, this appears to refer to the candidate image, and will be interpreted as such.
Claim 19, line 3: “the image” lacks antecedent basis, this also appears to refer to the candidate image, and will be interpreted as such.
Claims 2-14 are objected to due to their dependency on previously objected to claims.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 5-7 and 17-19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 5 and 17 both require “scoring the candidate image based at least in part on a similarity”, however, the specification does not describe “scoring” a candidate image in any words. The closest section is found in paragraph [0046], which describes “In some embodiments, the database may include images of fluids without defects and images of fluids with known defects. In some embodiments, the controller 106 is operable to identify a candidate impurity, or an air bubble, in the image based on a similarity between the image taken by the one or more cameras 108 and an image of the database. In some embodiments, an air bubble within the fluid may be identified as a candidate when a library image of an air bubble has similar characteristics to the image taken by camera 108.” This section describes identifying an impurity based on a similarity between a candidate image and an image or characteristics of an image of a known impurity. However, this section does not describe scoring a candidate image based on its similarity to the known impurity. Claims 6-7 and 18-19 are also rejected due to their dependence on a rejected claim.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 5-7, and 17-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “substantially” in claim 2 is a relative term which renders the claim indefinite. The term “substantially” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. It is unclear how perpendicular the first direction must be relative to the front side of the plurality of containers. Should the first direction be exactly perpendicular, forming two 90 degree angles between them, or is there some amount of tilt that would still render the first direction “substantially” perpendicular? As the claim currently reads, it is unclear.
Claims 5 and 17 both require “scoring the candidate image based at least in part on a similarity”, however, it is unclear what is meant by “scoring the candidate image.” Many different methods of scoring images are possible. For example, images could be ranked based on their similarity, images could be scored out of 10 (or any number for that matter) based on their similarity, or images could be given a percentage based on their similarity. Since there is insufficient written description to support this claim language, it is unclear what method would be used in this application. Claims 6-7 and 18-19 are also rejected due to their dependence on a rejected claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 20 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by “Automated Machine Vision System for Liquid Particle Inspection of Pharmaceutical Injection” (herein after referred to by its primary author, Zhang).
In regards to claim 20, Zhang teaches a fluid inspection system, comprising: a transportation system operable to transport a plurality of containers of fluid through an inspection zone (Zhang Figure 1 Outfeed, Carousel); a plurality of illuminators comprising: at least one front illuminator directed to the inspection zone (Zhang Figure 2(c) Rightmost LED); a back illuminator directed to an opposing direction of the at least one front illuminator (Zhang Figure 2(b) LED Light); and a transmission mode illuminator directed to the opposing direction of the at least one front illuminator (Zhang Figure 2(a) LED Light); one or more cameras directed to the inspection zone (Zhang Figures 2(a)-(c) CCD camera(s)); and a controller electrically coupled to the plurality of illuminators and the one or more cameras, the controller operable to: cause the back illuminator to emit pulses of light (Zhang Figure 2(b) LED Light); capture a plurality of images of the fluid in the plurality of containers disposed in the inspection zone, wherein the plurality of containers receive more light intensity from the back illuminator than from other illuminators illuminating a backside of the plurality of while the plurality of images are being captured; and identify an impurity in the fluid based on at least one image of the plurality of images (Zhang Figure 2(b) Description “light is sent through the container side. The camera looking from the other side of the container detects shadows created by the particle.”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-11, and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of JPH09325122A (herein after referred to by its primary author, Yamamoto).
In regards to claim 1, Zhang teaches a fluid inspection system, comprising: a transportation system operable to transport a plurality of containers of fluid through an inspection zone (Zhang Figure 1 Outfeed, Carousel); one or more cameras operable to view a front side of the plurality of containers from a first direction while the plurality of containers are in the inspection zone (Zhang Figure 2(a)-(c) CCD Camera(s)); a back illuminator positioned to direct light in a second direction on a backside of the plurality of containers while the plurality of containers are in the inspection zone (Zhang Figure 2(b) LED Light); and a controller operable to: cause the back illuminator to emit light that illuminates the backside of the plurality of containers in the second direction while the plurality of containers are in the inspection zone; and cause the one or more cameras to capture a plurality of images of the fluid in the plurality of containers while being illuminated by the back illuminator, wherein the plurality of containers receive more light intensity from the second direction than from other directions illuminating the backside of the plurality of while the plurality of images are being captured (Zhang Figure 2(b) Description “..light is sent through the container side. The camera looking from the other side of the container detects shadows created by the particle.”).
Zhang does not teach wherein the second direction oriented greater than 90 degrees and less than 180 degrees from the first direction.
However, Yamamoto teaches wherein the second direction oriented greater than 90 degrees and less than 180 degrees from the first direction (Yamamoto Figure 2; Page 3 Paragraph 4 “The light flux passes through the inside of the optical fiber 10 and is emitted toward the side surface of the container 1 as illumination light having an appropriate diffusion angle. Here, the light sources 81 and 82 are located on both sides of the container 1.”)
Yamamoto is considered to be analogous to the claimed invention because they are both in the same field of container inspection. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Zhang to include the teachings of Yamamoto, to provide the advantage of distinguishing air bubbles from impurities in the liquid (Yamamoto Page 2 Paragraph 6 “It is possible to distinguish between true foreign matter and bubbles in a transparent container. “)
In regards to claim 3, Zhang in view of Yamamoto teaches the fluid inspection system of claim 1. Zhang in view of Yamamoto does not explicitly teach wherein the second direction is oriented greater than 110 degrees and less than 160 degrees from the first direction (Yamamoto teaches an angle of ~110 degrees). However Zhang in view of Yamamoto does teach Yamamoto Page 2 Paragraph 7 “Therefore, according to the present invention, in order to satisfy all the above-mentioned conditions, the light source at a lateral position of the container irradiates the container and the liquid in the container with illumination light having a predetermined diffusion angle, and the above-mentioned illumination is performed. A camera is placed at a position where a large amount of light is reflected from a foreign object at a position offset by a certain angle from the direction perpendicular to the center of the light flux, and the camera allows the liquid inside the container to enter from the side of the container where illumination light does not enter or reflect.”, which suggests that the angle between the camera and illuminator should be positioned in such a way that a large amount of light is reflected from the container to the camera. It would have been obvious to one of ordinary skill in the art to arrive at the claimed range of angles as a result of routine optimization. One of ordinary skill in the art would have had a reasonable expectation of success to formulate the claimed angle range because Yamamoto Page 3, Paragraph 4 “The light flux passes through the inside of the optical fiber 10 and is emitted toward the side surface of the container 1 as illumination light having an appropriate diffusion angle. Here, the light sources 81 and 82 are located on both sides of the container 1. That is, it is provided at each position sandwiching the container 1 on the center line 14 passing through the center of the container 1. When the shutter 11 is opened, the respective illumination lights are projected toward the opposing outer peripheral surface of the container 1, and enter the inside of the container 1 based on the refraction index of air and glass and the refraction index of glass and liquid.”, which suggests that the angle of the illumination will depends on the refraction index of the container and the liquid which is within.
In regards to claim 4, Zhang in view of Yamamoto teaches the fluid inspection system of claim 1, wherein the controller is configured to: identify a candidate image from the plurality of images; and categorize an impurity in the fluid based on the candidate image (Zhang Figure 4 Examiner note: In this reference, a FAST detector is used for each image, and a corresponding candidate image of an impurity is obtained. The impurity is then tracked and identified based on the candidate image).
In regards to claim 5, Zhang in view of Yamamoto teaches the fluid inspection system of claim 4, wherein the controller is configured to score the candidate image based at least in part on a similarity between the candidate image and characteristics of a known impurity (Zhang Section IV C “Step 4: To select determinative features. Since the matrix Ac mentioned before is rich yet redundant, we design a classifier to extract determinative ones that best distinguish the foreground objects from the background. The classifier is defined as [Equation 22] where vector p represents the property of each template in the training set Ac, i.e., +1 for positive templates and −1 for negative templates. The solution of (22) is the sparse vector s, whose nonzero elements are denoted as the most discriminative features selected from the K-D feature space… Step 5: To measure confidence. The proposed SDC method has been developed based on the assumption that a target image region can be better represented by the sparse combination of positive templates, while a background patch can be better represented by the span of negative templates. An optimized matrix A’ with a dimensionality reduction of the training set is employed to be more discriminative.” Examiner note: The particles are classified, then a confidence score is given based on how well the particle represents the property of the templates in the training set Ac).
In regards to claim 6, Zhang in view of Yamamoto teaches the fluid inspection system of claim 5, wherein the controller is configured to identify a type of impurity in the candidate image (Zhang Figure 4 “Target tracking and classification”).
In regards to claim 7, Zhang in view of Yamamoto teaches the fluid inspection system of claim 6, wherein, responsive to a determination that the type of impurity comprises glass, signaling, by the controller, a sorting system to remove at least one container of the plurality of containers corresponding to the at least one image (Zhang Figure 4; Section II A “The turret output is according to the inspection results, where the container is addressed toward two outfeed levers and the rejected products are automatically collected in rejected trays.” Examiner note: Figure 4 shows that this defect detection system can detect glass particles, and section II A shows that when a defect is detected (such as glass) the corresponding container is rejected).
In regards to claim 8, Zhang in view of Yamamoto teaches the fluid inspection system of claim 1, comprising: at least one front illuminator directed to the inspection zone; and a transmission mode illuminator directed to an opposing direction of the at least one front illuminator (Zhang Figure 2(a) and 2(c) Examiner note: The rightmost LED light located in figure 2(c) is analogous to the front illuminator, and the LED light located in figure 2(a) is analogous to the transmission mode illuminator).
In regards to claim 9, Zhang in view of Yamamoto teaches the fluid inspection system of claim 8, wherein: the at least one front illuminator and the back illuminator comprise a reflection mode illuminator (Zhang Figure 2(c) Examiner note: As can best be understood by the examiner, a reflection mode illuminator directs light towards the inspection area for reflection into a camera. Figure 2(c) shows an LED (analogous to the front illuminator) emitting light onto the container, and the reflected light being pick up by the CCD camera.) (Yamamoto Figure 3 Examiner note: As can be seen on the bottom half of this figure, light rays sent from the light source 81 (analogous to the back illuminator) are reflected off the container and absorbed by camera 93.); and illumination of the transmission mode illuminator is on a same plane as the plurality of containers (Zhang Figure 2(a) LED Light).
In regards to claim 10, Zhang in view of Yamamoto teaches the fluid inspection system of claim 9, wherein at least one front illuminator comprises a first illuminator and a second illuminator directed to the inspection zone at an acute angle relative to a same plane of the transmission mode illuminator and the plurality of containers (Zhang Figure 2(c) Examiner note: As can be seen in the figure, the front illuminator is at a <90 degree angle from the plane passing perpendicular to the walls of the container. Furthermore, the illuminator is made up of a plurality of LED lights, each one could be considered an illuminator, and therefore the front illuminator comprises at least two illuminators).
In regards to claim 11, Zhang in view of Yamamoto teaches the fluid inspection system of claim 8, wherein positioning of the at least one front illuminator, the back illuminator, and the transmission mode illuminator form asymmetric illumination across the inspection zone (Zhang Figure 2(a) and 2(c) Examiner note: These illuminators are positioned in such a way that their illumination is not symmetric around any plane).
In regards to claim 15, Zhang in view of Yamamoto teaches a method for fluid inspection, the method comprising: directing light at a first direction through a container disposed in an inspection zone; capturing an image of a fluid in the container disposed in the inspection zone from a second direction (Zhang Figure 2(b) LED light and CCD camera), an included angle defined between the first direction and the second direction being greater than 90 degrees and less than 180 degrees (Yamamoto Figure 2; “The light flux passes through the inside of the optical fiber 10 and is emitted toward the side surface of the container 1 as illumination light having an appropriate diffusion angle. Here, the light sources 81 and 82 are located on both sides of the container 1”), wherein the container receives more light intensity from the first direction than from other directions illuminating a backside of the container while the image is being captured (Zhang Figure 2(b) ); and identifying an impurity in the fluid based on the image (Zhang Figure 2(b) description “light is sent through the container side. The camera looking from the other side of the container detects shadows created by the particle.”).
In regards to claim 16, Zhang in view of Yamamoto renders obvious the claim language as in the consideration of claim 4.
In regards to claim 17, Zhang in view of Yamamoto renders obvious the claim limitations as in the consideration of claim 5.
In regards to claim 18, Zhang in view of Yamamoto renders obvious the claim limitations as in the consideration of claim 6.
In regards to claim 19, Zhang in view of Yamamoto renders obvious the claim limitations as in the consideration of claim 7.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Yamamoto, and further in view of “Machine vision model for detection of foreign substances at the bottom of empty large volume parenteral” (herein after referred to by its primary author, Yuan)
In regards to claim 2, Zhang in view of Yamamoto teaches the fluid inspection system of claim 1, wherein: the first direction is oriented along a plane passing through the inspection zone and substantially perpendicular to the front side of the plurality of containers (Zhang Figure 2(b) CCD Camera); and no light illuminates the back of the plurality of containers from a direction under the plane while the plurality of images are being captured (Zhang Figure 2(b) Examiner note: The illumination of this disclosure is done on the same plane as the plurality of images, which therefore means the light is not illuminated from below the container).
Zhang in view of Yamamoto fails to teach wherein the back illuminator is positioned above the plane.
However, Yuan teaches wherein the back illuminator is positioned above the plane (Yuan Figure 2(f) Examiner note: The illumination of this figure is represented with red arrows. In this case, the left portion of the illuminator is above the plane the camera passes through).
Yuan is considered to be analogous to the claimed invention because they are both in the same field of container inspection. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Zhang in view of Yamamoto to include the teachings of Yuan, to provide the advantage of a system which can achieve clear imaging through light placement (Yuan Section 1 “The vision model can achieve clear imaging of the bottom under the condition of ensuring the inspection accuracy and inspection field of view, and it can avoid the interference of special form factor such as narrow bottle opening during the imaging process.”)
Claims 12-14 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Yamamoto, and further in view of US20140146370 (herein after referred to by its primary author, Banner)
In regards to claim 12, Zhang in view of Yamamoto teaches the fluid inspection system of claim 1, but fails to teach wherein the controller is operable to compare at least one image with images in a database.
However, Banner teaches wherein the controller is operable to compare at least one image with images in a database (Banner Paragraph [0038] “Following registration, defect detection can be performed on the registered reference and target images. A defect detection function can implement a structural similarity information measure (SSIM) on a pixel-by-pixel basis for example, although other suitable processes for determining the presence of defects are available.”).
Banner is considered to be analogous to the claimed invention because they are both in the same field of defect detection. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Zhang in view of Yamamoto to include the teachings of Banner, to provide the advantage of defect detection which is most similar to human checking and considers every portion of the target and reference image (Banner Paragraph [0038] “The SSIM is based on the precept that every region in the target image should have a similar region nearby in the reference image, unless it contains a defect and that human visual perception is highly adapted for extracting structural information from an image.”)
In regards to claim 13, Zhang in view of Yamamoto and Banner teaches the fluid inspection system of claim 12, wherein the controller is operable to identify a candidate impurity in the at least one image based on a similarity between the at least one image and at least one image of the database (Banner Paragraph [0038] “Following registration, defect detection can be performed on the registered reference and target images. A defect detection function can implement a structural similarity information measure (SSIM) on a pixel-by-pixel basis for example, although other suitable processes for determining the presence of defects are available.”).
In regards to claim 14, Zhang in view of Yamamoto and Banner teaches the fluid inspection system of claim 13, wherein the candidate impurity is identified via a structure similarity index measurement performed by the controller (Banner Paragraph [0038] “Following registration, defect detection can be performed on the registered reference and target images. A defect detection function can implement a structural similarity information measure (SSIM) on a pixel-by-pixel basis for example, although other suitable processes for determining the presence of defects are available.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US20200408702 teaches a method of visually inspecting a container with liquid using various illumination methods.
“Intelligent Foreign Particle Inspection Machine for Injection Liquid Examination Based on Modified Pulse-Coupled Neural Networks” teaches a method of detecting foreign matter (impurities) within liquid containers.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677