DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/22/2024 has been considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 17 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 17 the phrase "and/or" renders the claims indefinite because it is unclear whether the limitations following the claims are part of the claimed invention. See MPEP 2173.05(d). For purposes of examination, the limitation will be read as "or".
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 6, 12-15, 17-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”).
Regarding claim 1, Odhner discloses a computer-implemented method for detecting empty container (Abstract “Devices, systems, and methods for determining whether a container is empty in the context of robotic picking solutions.”; Paragraph [0081] “are described above with reference to block diagrams and/or operational illustrations of methods, systems, and computer program products according to embodiments of the present disclosure”), the method comprising:
acquiring, using an imaging system, a first image (container data in Paragraph [0054] equates to first image) of a container (Paragraph [0054] “the processing algorithms may generate a confidence score regarding the likelihood that the container is empty based on received container data. In some cases, the processing algorithm may generate only a low-confidence estimate of the state of the container (i.e., whether or not the container is empty). This may occur when small, lightweight objects are near container edges or corners, or when there is insufficient lighting”; Paragraph [0065] "The system may include a set of sensors (e.g., scales, range cameras, RGB cameras) or the like, and execute novel processing algorithms to fuse data from these sensor modalities to determine if a container is empty as discussed above. The sensors used may include any type of cameras (e.g., visible spectrum, infrared, hyperspectral, depth, etc.) that are statically mounted on or attached to an end effector (i.e., a hand portion) of the picking device.");
after the first image is acquired, causing a perturbation device to perturb a content of the container (Paragraph [0054] “when the processing algorithms are not confident regarding whether the container is empty (e.g., an outputted confidence value is below some threshold), the perturbation device 130 of the robotic picking device 102 may perturb the container.”; Paragraph [0055] "The picking device 102 may grasp items in the container by itself or in conjunction with the perturbation device 130 or the transportation device 120.”);
acquiring, using the imaging system, a second image (reacquired data after the container is perturbed in Paragraph [0056] equates to second image data) of the container after the perturbation (Paragraph [0056] "the container may be perturbed from its side or bottom by a perturbation device 130. Once the container is perturbed, the appropriate sensors can reacquire data and the processing algorithms can execute again to generate a score. This process may iterate in some embodiments until a confident determination is made."); and
processing, using one or more computer processors, the first image (container data in Paragraph [0054] equates to first image) and the second image (second set of data after the container is perturbed in Paragraph [0057] equates to second image data) to determine whether the container is empty [based on whether there is a difference between the first image and the second image] as a result of one or more objects inside the container being moved due to the perturbation (Paragraph [0051] "a number of image processing methods can be employed to detect items in the container that are otherwise undetectable."; Paragraph [0054] “the processing algorithms may generate a confidence score regarding the likelihood that the container is empty based on received container data. In some cases, the processing algorithm may generate only a low-confidence estimate of the state of the container (i.e., whether or not the container is empty); Paragraph [0057] "if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data...the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container...If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty.").
However, Odhner fails to teach based on whether there is a difference between the first image and the second image.
Huang teaches based on whether there is a difference between the first image (current image in Paragraph [0027 equates to the first image) and the second image (forward image in Paragraph [0027] equates to the second image) (Paragraph [0027] “comparing the average pixel difference of each region in the current image with the average pixel difference of the corresponding position region in the forward image, and judging whether shear exists in the current image or not by combining an empirical threshold.”; Paragraph [0032] “the average pixel difference of each image region between adjacent frames is compared (average absolute difference, mean square error, etc. can be used), and an empirical threshold is combined to determine whether shear exists in the current image (for example, the minimum value of the average absolute difference of each region can be compared with the empirical threshold)”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner’s reference to include based on whether there is a difference between the first image and the second image taught by Huang’s reference. The motivation for doing so would have been to analyze the consistency of the motion states of the current area and the reference area in the motion trail direction as suggested by Huang (see Huang, Paragraph [0037]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Huang with Odhner to obtain the invention specified in claim 1.
Regarding claim 3, which claim 1 is incorporated, Odhner discloses wherein the perturbation is performed by moving a robot end effector along an inside of the container in a stirring motion (Paragraph [0077] "The robotic picking device 600 may perturb an item in the container 606 by lowering the head portion 602 into the container 606, and moving the head 602 horizontally or vertically therein.").
Regarding claim 4, which claim 1 is incorporated, Odhner discloses wherein the perturbation is performed by shaking or tilting the container (Paragraph [0077] "Grips of the head portion 602 may also grasp the edge of the container 606 and shake the container 606 to determine if items remain in the container 606.").
Regarding claim 6, which claim 1 is incorporated, Odhner discloses wherein processing the first image and the second image comprises (Paragraph [0057] "if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data. As discussed above, the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container. The system 100 may then calculate a second confidence score. If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty."):
[comparing the first image and the second image by evaluating a value of a mean squared error (MSE) function between the first image and the second image] (Paragraph [0068] “values from one or more of these sensors may be combined using various algorithms to determine whether the container is empty.”);
determining that the container is empty upon [determining that the value of the MSE function is less than a pre-defined threshold value] (Paragraph [0076] " the processor may generate a confidence score higher than a predetermined threshold. A confidence score higher than a predetermined threshold may indicate that the container is empty (Step 506)."); and
determining that the container is not empty upon [determining that the value of the MSE function is equal to or greater than the pre-defined threshold value] (Paragraph [0078] " the processor may generate a confidence score that is lower than a predetermined threshold. A confidence score lower than a predetermined threshold may indicate that the container is not empty or that the system executing the method shown in FIG. 5 cannot reliably determine that the container is empty.”).
However, Odhner fails to teach comparing the first image and the second image by evaluating a value of a mean squared error (MSE) function between the first image and the second image; determining that the value of the MSE function is less than a pre-defined threshold value; and determining that the value of the MSE function is equal to or greater than the pre-defined threshold value.
Huang teaches comparing the first image and the second image by evaluating a value of a mean squared error (MSE) function between the first image and the second image (Paragraph [0027] “comparing the average pixel difference of each region in the current image with the average pixel difference of the corresponding position region in the forward image, and judging whether shear exists in the current image or not by combining an empirical threshold.”; Paragraph [0032] “the average pixel difference of each image region between adjacent frames is compared (average absolute difference, mean square error, etc. can be used), and an empirical threshold is combined to determine whether shear exists in the current image (for example, the minimum value of the average absolute difference of each region can be compared with the empirical threshold)”);
determining that the value of the MSE function is less than a pre-defined threshold value (Paragraph [0033] “a down-sampling operation may be performed on the current image and the forward image, and then, for each region in the down-sampled image, the average pixel difference between the down-sampled image and the corresponding region in the forward direction is compared, and whether shear exists in the current down-sampled image is determined by combining an empirical threshold, so as to determine whether shear exists in the current image”; Paragraph [0034] “in the absence of scene shear, conventional three-field or five-field motion estimation may be selected.”); and
determining that the value of the MSE function is equal to or greater than the pre-defined threshold value (Paragraph [0034] “Taking a video de-interlacing method as an example, the specific implementation of the motion estimation stage is as follows: when scene shear exists, the time domain range of three-dimensional recursive search is limited”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner’s reference to include comparing the first image and the second image by evaluating a value of a mean squared error (MSE) function between the first image and the second image; determining that the value of the MSE function is less than a pre-defined threshold value; and determining that the value of the MSE function is equal to or greater than the pre-defined threshold value taught by Huang’s reference. The motivation for doing so would have been to analyze the consistency of the motion states of the current area and the reference area in the motion trail direction as suggested by Huang (see Huang, Paragraph [0037]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Huang with Odhner to obtain the invention specified in claim 6.
Regarding claim 12, which claim 1 is incorporated, Odhner discloses upon determining that the container is empty, reporting to an automated storage and retrieval system (AS/RS) that the container is empty (Paragraph [0058] "The system 100 may move a container from one location to another upon determining that the container is empty. For example, if the container is on a conveyor belt and the system determines that the container is empty, the system may control the conveyor belt to move the container to a storage location. In some embodiments, the system 100 may move the empty container to a second conveyor belt or to a second location to be re-filled with more items.").
Regarding claim 13, Odhner discloses a system for detecting empty container (Paragraph [0015] “embodiments relate to a system for detecting whether a container is empty.”), the system comprising:
a perturbation device for perturbing a content of a container (Paragraph [0016] “the system further includes a picking device to perturb the container.”;
an imaging system for acquiring a first image and a second image of the container (Paragraph [0015] “The system includes a plurality of sensors configured to gather container data regarding a container at a first location, the container data including at least two of weight data related to the container, depth data related to the container, and color sensor data related to the container”; Paragraph [0065] "The system may include a set of sensors (e.g., scales, range cameras, RGB cameras) or the like, and execute novel processing algorithms to fuse data from these sensor modalities to determine if a container is empty as discussed above. The sensors used may include any type of cameras (e.g., visible spectrum, infrared, hyperspectral, depth, etc.) that are statically mounted on or attached to an end effector (i.e., a hand portion) of the picking device."), the first image (container data in Paragraph [0054] equates to first image) being acquired before the perturbation (Paragraph [0054] “ the processing algorithms may generate a confidence score regarding the likelihood that the container is empty based on received container data. In some cases, the processing algorithm may generate only a low-confidence estimate of the state of the container (i.e., whether or not the container is empty). This may occur when small, lightweight objects are near container edges or corners, or when there is insufficient lighting”; Paragraph [0065] "The system may include a set of sensors (e.g., scales, range cameras, RGB cameras) or the like, and execute novel processing algorithms to fuse data from these sensor modalities to determine if a container is empty as discussed above. The sensors used may include any type of cameras (e.g., visible spectrum, infrared, hyperspectral, depth, etc.) that are statically mounted on or attached to an end effector (i.e., a hand portion) of the picking device."), and the second image (reacquired data after the container is perturbed in Paragraph [0056] equates to second image data) being acquired after the perturbation (Paragraph [0056] "the container may be perturbed from its side or bottom by a perturbation device 130. Once the container is perturbed, the appropriate sensors can reacquire data and the processing algorithms can execute again to generate a score. This process may iterate in some embodiments until a confident determination is made."); and
one or more computer processors configured to process the first image (container data in Paragraph [0054] equates to first image) and the second image (second set of data after the container is perturbed in Paragraph [0057] equates to second image data) to determine whether the container is empty [based on whether there is a difference between the first image and the second image] as a result of one or more objects inside the container being moved due to the perturbation (Paragraph [0051] "a number of image processing methods can be employed to detect items in the container that are otherwise undetectable."; Paragraph [0054] “the processing algorithms may generate a confidence score regarding the likelihood that the container is empty based on received container data. In some cases, the processing algorithm may generate only a low-confidence estimate of the state of the container (i.e., whether or not the container is empty); Paragraph [0057] "if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data...the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container...If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty.").
However, Odhner fails to teach based on whether there is a difference between the first image and the second image.
Huang teaches based on whether there is a difference between the first image (current image in Paragraph [0027 equates to the first image) and the second image (forward image in Paragraph [0027] equates to the second image) (Paragraph [0027] “comparing the average pixel difference of each region in the current image with the average pixel difference of the corresponding position region in the forward image, and judging whether shear exists in the current image or not by combining an empirical threshold.”; Paragraph [0032] “the average pixel difference of each image region between adjacent frames is compared (average absolute difference, mean square error, etc. can be used), and an empirical threshold is combined to determine whether shear exists in the current image (for example, the minimum value of the average absolute difference of each region can be compared with the empirical threshold)”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner’s reference to include based on whether there is a difference between the first image and the second image taught by Huang’s reference. The motivation for doing so would have been to analyze the consistency of the motion states of the current area and the reference area in the motion trail direction as suggested by Huang (see Huang, Paragraph [0037]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Huang with Odhner to obtain the invention specified in claim 13.
Regarding claim 14. Which claim 13 is incorporated, Odhner discloses wherein the imaging system comprises one or more cameras, or one or more ultrasonic sensors, or one or more radars, or one or more lidars, or a combination thereof (Paragraph [0065] "The system may include a set of sensors (e.g., scales, range cameras, RGB cameras) or the like, and execute novel processing algorithms to fuse data from these sensor modalities to determine if a container is empty as discussed above. The sensors used may include any type of cameras (e.g., visible spectrum, infrared, hyperspectral, depth, etc.) that are statically mounted on or attached to an end effector (i.e., a hand portion) of the picking device.").
Regarding claim 15, which claim 13 is incorporated, Odhner discloses wherein the imaging system comprises two RGBD cameras (Paragraph [0065] "The system may include a set of sensors (e.g., scales, range cameras, RGB cameras) or the like, and execute novel processing algorithms to fuse data from these sensor modalities to determine if a container is empty as discussed above. The sensors used may include any type of cameras (e.g., visible spectrum, infrared, hyperspectral, depth, etc.) that are statically mounted on or attached to an end effector (i.e., a hand portion) of the picking device.").
Regarding claim 17, which claim 13 is incorporated, Odhner discloses wherein the perturbation device comprises a robot end effector configured to be moved along an inside of the container in a stirring motion, and/or a shaker for shaking the container, and/or a tilting stage for tilting the container (Paragraph [0077] "The robotic picking device 600 may perturb an item in the container 606 by lowering the head portion 602 into the container 606, and moving the head 602 horizontally or vertically therein. Grips of the head portion 602 may also grasp the edge of the container 606 and shake the container 606 to determine if items remain in the container 606.").
Regarding claim 18 (drawn to a system) claim 18 is rejected the same as claim 6 and the arguments similar to that presented above for claim 6 are equally applicable to the claim 18, and all the other limitations similar to claim 6 are not repeated herein, but incorporated by reference.
Regarding claim 20, which claim 1 is incorporated, Odhner discloses a tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more hardware processors, alone or in combination, provide for execution of the method of claim 1 (Paragraph [0015] “The system includes a plurality of sensors configured to gather container data regarding a container at a first location, the container data including at least two of weight data related to the container, depth data related to the container, and color sensor data related to the container; and a processor configured to execute instructions stored on memory to provide a sensor fusion module configured to process the received container data to determine whether the container is empty.”).
Claims 2 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”) and further in view of Duan (US 2021/0069903 A1).
Regarding claim 2, which claim 1 is incorporated, Odhner and Huang both fail to teach wherein the perturbation is performed by blowing air inside the container.
Duan teaches wherein the perturbation is performed by blowing air inside the container (Paragraph [0024] “the blowing perturbation element may take a general, non-targeted strategy wherein it continuously blows air around the inner perimeter of the bin.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include wherein the perturbation is performed by blowing air inside the container taught by Duan’s reference. The motivation for doing so would have been to effectively disturb the position of an object in the bin as suggested by Duan (see Duan, Column Paragraph [0024]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Duan with Odhner and Huang to obtain the invention specified in claim 2.
Regarding claim 16, which claim 13 is incorporated, Odhner and Huang both fail to teach wherein the perturbation device comprises a nozzle for blowing air inside the container.
Duan teaches wherein the perturbation device comprises a nozzle for blowing air inside the container (Paragraph [0032] “Perturbation element 120, in the present example, is a pneumatic air valve connected to a pneumatic air supply, wherein the pneumatic air valve blows compressed air into bin 125 in certain situations.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include wherein the perturbation device comprises a nozzle for blowing air inside the container taught by Duan’s reference. The motivation for doing so would have been to effectively disturb the position of an object in the bin as suggested by Duan (see Duan, Column Paragraph [0024]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Duan with Odhner and Huang to obtain the invention specified in claim 16.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”), and further in view of Bonewitz et al. (US 5,917,602 A)
Regarding claim 5, which claim 1 is incorporated, Odhner and Huang both fail to teach wherein the first image and the second image are acquired under a same set of environmental conditions.
Bonewitz teaches wherein the first image and the second image are acquired under a same set of environmental conditions (Column 5 [lines 34-39] “The lighting assembly 160 illuminates containers 114 as conveyor 120 moves them through the imaging area in the optical path of camera 158. Preferably, lighting assembly 160 has a fluorescent light source 162 which provides relatively constant and even illumination of container 114.”)
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include wherein the first image and the second image are acquired under a same set of environmental conditions taught by Bonewitz’s reference. The motivation for doing so would have been to cause higher contrast edges in the acquired image and improve profile inspection as suggested by Bonewitz (see Bonewitz, Column 9 [lines 46-48]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Bonewitz with Odhner and Huang to obtain the invention specified in claim 5.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”), and further in view of Raghoebardajal et al (US 2011/0064375 A1) (hereinafter, “Raghoebardajal”).
Regarding claim 7, which claim 6 is incorporated, Odhner discloses wherein processing the first image and the second image [further comprises, before evaluating the value of the MSE function, applying a blur function to the first image and the second image] (Paragraph [0057] "if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data. As discussed above, the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container. The system 100 may then calculate a second confidence score. If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty.").
However; Odhner fails to teach before evaluating the value of the MSE function, applying a blur function to the first image and the second image.
Huang teaches evaluating the value of the MSE function (Paragraph [0032] “the average pixel difference of each image region between adjacent frames is compared (average absolute difference, mean square error, etc. can be used), and an empirical threshold is combined to determine whether shear exists in the current image (for example, the minimum value of the average absolute difference of each region can be compared with the empirical threshold)”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner’s reference to include evaluating the value of the MSE function taught by Huang’s reference. The motivation for doing so would have been to analyze the consistency of the motion states of the current area and the reference area in the motion trail direction as suggested by Huang (see Huang, Paragraph [0037]).
However, Odhner and Huang both fail to teach before [evaluating the value of the MSE function,] applying a blur function to the first image and the second image.
Raghoebardajal teaches before [evaluating the value of the MSE function,] applying a blur function to the first image and the second image (Paragraph [0067] “image pre-processing operations such as blur reduction, noise filtering and the like may be carried out by the cell processor 100 on the image frames before the cell processor 100 carries out detection of inter-image motion.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include before [evaluating the value of the MSE function,] applying a blur function to the first image and the second image taught by Raghoebardajal’s reference. The motivation for doing so would have been to reduce the effects of lighting differences, pixel noise, and the like that lead to false detection of motion as suggested by Raghoebardajal (see Raghoebardajal, Paragraph [0067]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Raghoebardajal with Odhner and Huang to obtain the invention specified in claim 7.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”), and further in view of Hong et al. (US 2021/0021833 A1) (hereinafter, “Hong”).
Regarding claim 8, which claim 6 is incorporated, Odhner discloses wherein processing the first image and the second image [further comprises, before evaluating the value of the MSE function, converting the first image and the second image into greyscale images] (Paragraph [0057] "if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data. As discussed above, the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container. The system 100 may then calculate a second confidence score. If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty.").
However; Odhner fails to teach before evaluating the value of the MSE function, converting the first image and the second image into greyscale images.
Huang teaches evaluating the value of the MSE function (Paragraph [0032] “the average pixel difference of each image region between adjacent frames is compared (average absolute difference, mean square error, etc. can be used), and an empirical threshold is combined to determine whether shear exists in the current image (for example, the minimum value of the average absolute difference of each region can be compared with the empirical threshold)”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner’s reference to include evaluating the value of the MSE function taught by Huang’s reference. The motivation for doing so would have been to analyze the consistency of the motion states of the current area and the reference area in the motion trail direction as suggested by Huang (see Huang, Paragraph [0037]).
However, Odhner and Huang both fail to teach before [evaluating the value of the MSE function,] converting the first image and the second image into greyscale images.
Hong teaches before [evaluating the value of the MSE function,] converting the first image and the second image into greyscale images (Paragraph [0044] “Grayscale frames can be used to detect if there is motion, so color video frames can be converted into grayscale frames. For example, a 64×48 Bayer frame is obtained and downsampled. The Bayer frame can be converted into grayscale image by averaging the R, Gr, Gb, B channels.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include before [evaluating the value of the MSE function,] converting the first image and the second image into greyscale images taught by Hong’s reference. The motivation for doing so would have been to help speed up processing time and reduce memory usage as suggested by Hong (see Hong, Paragraph [0042]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Hong with Odhner and Huang to obtain the invention specified in claim 8.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”), and further in view of Chen et al. (US 2017/0188038 A1) (hereinafter, “Chen”).
Regarding claim 9, which claim 1 is incorporated, Odhner discloses wherein processing the first image and the second image comprises (Paragraph [0057] " if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data. As discussed above, the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container. The system 100 may then calculate a second confidence score. If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty."):
[comparing the first image and the second image by evaluating a value of a structural similarity index measure (SSIM) function between the first image and the second image] (Paragraph [0068] “values from one or more of these sensors may be combined using various algorithms to determine whether the container is empty.”);
determining that the container is empty upon [determining that a difference between the value of the SSIM function and unity is less than a pre-defined threshold amount] (Paragraph [0076] " the processor may generate a confidence score higher than a predetermined threshold. A confidence score higher than a predetermined threshold may indicate that the container is empty (Step 506)."); and
determining that the container is not empty upon [determining that the difference between the value of the SSIM function and unity is equal to or greater than the pre-defined threshold amount] (Paragraph [0078] " the processor may generate a confidence score that is lower than a predetermined threshold. A confidence score lower than a predetermined threshold may indicate that the container is not empty or that the system executing the method shown in FIG. 5 cannot reliably determine that the container is empty.”).
However, Odhner fails to teach comparing the first image and the second image by evaluating a value of a structural similarity index measure (SSIM) function between the first image and the second image; determining that a difference between the value of the SSIM function and unity is less than a pre-defined threshold amount; and determining that the difference between the value of the SSIM function and unity is equal to or greater than the pre-defined threshold amount.
Huang teaches comparing the first image and the second image by evaluating a value of a [structural similarity index measure (SSIM)] function between the first image and the second image (Paragraph [0027] “comparing the average pixel difference of each region in the current image with the average pixel difference of the corresponding position region in the forward image, and judging whether shear exists in the current image or not by combining an empirical threshold.”; Paragraph [0032] “the average pixel difference of each image region between adjacent frames is compared (average absolute difference, mean square error, etc. can be used), and an empirical threshold is combined to determine whether shear exists in the current image (for example, the minimum value of the average absolute difference of each region can be compared with the empirical threshold)”);
determining that a difference between the value [of the SSIM function and unity] is less than a pre-defined threshold amount (Paragraph [0033] “a down-sampling operation may be performed on the current image and the forward image, and then, for each region in the down-sampled image, the average pixel difference between the down-sampled image and the corresponding region in the forward direction is compared, and whether shear exists in the current down-sampled image is determined by combining an empirical threshold, so as to determine whether shear exists in the current image”; Paragraph [0034] “in the absence of scene shear, conventional three-field or five-field motion estimation may be selected.”); and
determining that the difference between the value [of the SSIM function and unity] is equal to or greater than the pre-defined threshold amount (Paragraph [0034] “Taking a video de-interlacing method as an example, the specific implementation of the motion estimation stage is as follows: when scene shear exists, the time domain range of three-dimensional recursive search is limited”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner’s reference to include comparing the first image and the second image by evaluating a value of a [structural similarity index measure (SSIM)] function between the first image and the second image; determining that a difference between the value [of the SSIM function and unity] is less than a pre-defined threshold amount; and determining that the difference between the value [of the SSIM function and unity] is equal to or greater than the pre-defined threshold amount taught by Huang’s reference. The motivation for doing so would have been to analyze the consistency of the motion states of the current area and the reference area in the motion trail direction as suggested by Huang (see Huang, Paragraph [0037]).
However, Odhner and Huang both fail to teach a structural similarity index measure (SSIM) and unity.
Chen teaches a structural similarity index measure (SSIM) and unity (Paragraph [0045] “The similarity calculation circuit 132 is configured to perform the similarity calculation on the target patch 330 and the first patch 310 to obtain a motion value γ…The motion value γ is indicative of the similarity level. In the present embodiment, the similarity calculation may include the temporal difference, the temporal and spatial difference, the optical flow, the cross correlation, the structural similarity SSIM”; Paragraph [0046] “the larger the motion value γ is, the more second patches 320 are added into the set of third patches 340, and the smaller the motion value γ is, the less second patches are added into the set of third patches.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include a structural similarity index measure (SSIM) and unity taught by Chen’s reference. The motivation for doing so would have been to obtain a motion value that is indicative of the similarity between a patch and a target patch as suggested by Chen (see Chen, Paragraph [0045]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Chen with Odhner and Huang to obtain the invention specified in claim 9.
Regarding claim 19 (drawn to a system) claim 19 is rejected the same as claim 9 and the arguments similar to that presented above for claim 9 are equally applicable to the claim 19, and all the other limitations similar to claim 9 are not repeated herein, but incorporated by reference.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”), further in view of Chen et al. (US 2017/0188038 A1) (hereinafter, “Chen”) and Raghoebardajal et al (US 2011/0064375 A1) (hereinafter, “Raghoebardajal”).
Regarding claim 10, which claim 9 is incorporated, Odhner discloses wherein processing the first image and the second image [further comprises, before evaluating the value of the SSIM function, applying a blur function to the first image and the second image] (Paragraph [0057] "if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data. As discussed above, the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container. The system 100 may then calculate a second confidence score. If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty.").
However, Odhner and Huang both fail to teach before evaluating the value of the SSIM function, applying a blur function to the first image and the second image.
Chen teaches evaluating the value of the SSIM function (Paragraph [0045] “The similarity calculation circuit 132 is configured to perform the similarity calculation on the target patch 330 and the first patch 310 to obtain a motion value γ…The motion value γ is indicative of the similarity level. In the present embodiment, the similarity calculation may include the temporal difference, the temporal and spatial difference, the optical flow, the cross correlation, the structural similarity SSIM”; Paragraph [0046] “the larger the motion value γ is, the more second patches 320 are added into the set of third patches 340, and the smaller the motion value γ is, the less second patches are added into the set of third patches.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include evaluating the value of the SSIM function taught by Chen’s reference. The motivation for doing so would have been to obtain a motion value that is indicative of the similarity between a patch and a target patch as suggested by Chen (see Chen, Paragraph [0045]).
However, Odhner, Huang, and Chen fail to teach before [evaluating the value of the SSIM function,] applying a blur function to the first image.
Raghoebardajal teaches before [evaluating the value of the SSIM function,] applying a blur function to the first image and the second image (Paragraph [0067] “image pre-processing operations such as blur reduction, noise filtering and the like may be carried out by the cell processor 100 on the image frames before the cell processor 100 carries out detection of inter-image motion.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang, and further in view of Chen to include before [evaluating the value of the SSIM function,] applying a blur function to the first image and the second image taught by Raghoebardajal’s reference. The motivation for doing so would have been to reduce the effects of lighting differences, pixel noise, and the like that lead to false detection of motion as suggested by Raghoebardajal (see Raghoebardajal, Paragraph [0067]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Raghoebardajal with Odhner, Huang, and Chen to obtain the invention specified in claim 10.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Odhner et al. (US 2022/0032467 A1) (hereinafter, “Odhner”) in view of Huang et al. (CN 107,333,174 B) (hereinafter, “Huang”), further in view of Chen et al. (US 2017/0188038 A1) (hereinafter, “Chen”) and Hong et al. (US 2021/0021833 A1) (hereinafter, “Hong”).
Regarding claim 11, which claim 9 is incorporated, Odhner discloses wherein processing the first image and the second image [further comprises, before evaluating the value of the SSIM function, converting the first image and the second image into greyscale images] (Paragraph [0057] "if the calculated confidence score is below the predetermined threshold, the system 100 may gather a second set of data. As discussed above, the system 100 may perturb the container with a perturbation device 130 or the transportation device 120 and then gather a second set of data regarding the perturbed container. The system 100 may then calculate a second confidence score. If the second confidence score is higher than the predetermined threshold, the system 100 may determine that the container is empty.").
However; Odhner and Huang both fail to teach before evaluating the value of the SSIM function, converting the first image and the second image into greyscale images.
Chen teaches evaluating the value of the SSIM function (Paragraph [0045] “The similarity calculation circuit 132 is configured to perform the similarity calculation on the target patch 330 and the first patch 310 to obtain a motion value γ…The motion value γ is indicative of the similarity level. In the present embodiment, the similarity calculation may include the temporal difference, the temporal and spatial difference, the optical flow, the cross correlation, the structural similarity SSIM”; Paragraph [0046] “the larger the motion value γ is, the more second patches 320 are added into the set of third patches 340, and the smaller the motion value γ is, the less second patches are added into the set of third patches.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang to include evaluating the value of the SSIM function taught by Chen’s reference. The motivation for doing so would have been to obtain a motion value that is indicative of the similarity between a patch and a target patch as suggested by Chen (see Chen, Paragraph [0045]).
However, Odhner, Huang, and Cheng fail to teach before [evaluating the value of the SSIM function,] converting the first image and the second image into greyscale images.
Hong teaches before [evaluating the value of the SSIM function,] converting the first image and the second image into greyscale images (Paragraph [0044] “Grayscale frames can be used to detect if there is motion, so color video frames can be converted into grayscale frames. For example, a 64×48 Bayer frame is obtained and downsampled. The Bayer frame can be converted into grayscale image by averaging the R, Gr, Gb, B channels.”).
Therefore, it would have been obvious to one of ordinary skill of the art before the effective filing date to modify Odhner in view of Huang, and further in view of Chen to include before [evaluating the value of the SSIM function,] converting the first image and the second image into greyscale images taught by Hong’s reference. The motivation for doing so would have been to help speed up processing time and reduce memory usage as suggested by Hong (see Hong, Paragraph [0042]).
Further, one skilled in the art could have combined the elements described above by known methods with no change to the respective functions, and the combination would have yielded nothing more that predictable results. Therefore, it would have been obvious to combine Hong with Odhner, Huang, and Chen to obtain the invention specified in claim 11.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chen et al. (US 2022/0222799 A1) discloses a method of detecting defect in images of finished products by obtaining an image to be tested, a reconstructed image and an error to be tested.
Osunkwo et al. (US 2023/0053085 A1) discloses a system comprising a defect detection model that compares an input image to a template image to identify defects.
Gomm et al. (US 4,136,930 A) discloses acquiring two images of a bottle before and after rotating the bottle and comparing the images to detect movement indicative of foreign particles within the liquid.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UROOJ FATIMA whose telephone number is (571)272-2096. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UROOJ FATIMA/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights