DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 51 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter as follows: Claim 51 is drawn to a “computer program product” which is essentially a program/software per se. Software do not fall within the definition of a process, machine, manufacture, or composition of matter (In re Nuijten), and are therefore non-statutory. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 32-42 and 48-51 are rejected under 35 U.S.C. 103 as being unpatentable over Horowitz et al. US2019/0130560 hereinafter referred to as Horowitz in view of Sun et al. US2017/0206431 hereinafter referred to as Sun.
As per Claim 32, Horowitz teaches a method for determining whether at least one predetermined transport item is arranged in a monitoring area, (Horowitz, Figure 1)comprising:
acquiring an image signal of the monitoring area through which a transport path of an object passes is acquired; (Horowitz, Paragraph [0018], “In the embodiment shown in FIG. 1, the imaging device 162 functions as an area sensor that captures a two dimensional image (also referred to herein as an image “frame”) of an area 60 of the conveyor belt 50, along with any objects 55 that are within that area 60. In other embodiments, the imaging device 162 may comprise a line scanner that captures a strip or slice of the conveyor belt 50 and its contents. These image strips may then be digitally spliced to generate a composite image frame of area 60”)
supplying the image signal to an artificial neural network, (Horowitz, Paragraph [0024], “In operation the imaging device 162 is directed downward towards the conveyor belt 50 in order to capture an overhead view of the materials 55 being transported by the conveyor belt 50. The imaging device 162 produces an image signal that is delivered to the object characterization processor 160. Within the object characterization processor, these image frames are provided input to one or more neural network and artificial intelligence algorithms (shown as the Neural Processing Units 164) to locate and identify material appealing within the image frames”)
generating an image of the monitoring that at least a part of the at least one predetermined transport item is arranged in the monitoring area. (Horowitz, Paragraph [0018], “In the embodiment shown in FIG. 1, the imaging device 162 functions as an area sensor that captures a two dimensional image (also referred to herein as an image “frame”) of an area 60 of the conveyor belt 50, along with any objects 55 that are within that area 60. Horowitz does not explicitly teach wherein before supplying the image signal to the artificial neural network, it is determined by another artificial neural network based on the image signal whether at least one part of an object is arranged in the monitoring area,
wherein the image signal is supplied to the artificial neural network when it is determined by the other artificial neural network that at least a part of the object is arranged in the monitoring area; determining by the artificial neural network based on the image signal whether the at least one part of the object corresponds to at least a part of the at least one predetermined transport item; and
Sun teaches wherein before supplying the image signal to the artificial neural network, it is determined by another artificial neural network based on the image signal whether at least one part of an object is arranged in the monitoring area, (Sun, Claim 1, “ identifying, by a first type of neural network, a candidate object in the input image;”)
wherein the image signal is supplied to the artificial neural network when it is determined by the other artificial neural network that at least a part of the object is arranged in the monitoring area; determining by the artificial neural network based on the image signal whether the at least one part of the object corresponds to at least a part of the at least one predetermined transport item; and (Sun, Claim 1, “determining, by a second type of neural network, a category of the candidate object”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Sun into Horowitz because by replacing the neural network of Horowitz with a dual neural network system of Sun will increase the efficiency and minimize processing power to produce classification results.
Therefore it would have been obvious to one of ordinary skill to combine the two references to obtain the invention in Claim 32.
As per Claim 33, Horowitz in view of Sun teaches the method according to claim 32, wherein the artificial neural network has a convolutional neural network, wherein: a. the image signal is supplied to an input layer of the convolutional neural network; and/or b. a number of neurons of an input layer of the convolutional neural network corresponds to a number of pixels of the image signal; and/or c. an input layer of the convolutional neural network is three-dimensional. (Sun, Paragraph [0043], “In some examples, the initial processing module can include a Deep Convolutional Neural Network (Deep CNN). In such examples, the Deep CNN can process the input image through multiple convolutional layers, maximum pooling layers, region of interest pooling layers, and/or fully-connected layers”)
The rationale applied to the rejection of claim 32 has been incorporated herein.
As per Claim 34, Horowitz in view of Sun teaches the method according to claim 33, wherein: a. the artificial neural network has at least one layer, wherein the at least one layer is a neural convolutional layer, and/or wherein the at least one layer is connected to the input layer and/or receives output data from the input layer; and/or b. the convolutional neural network has multiple layers, each having one or more sub-layers, wherein a first layer follows the input layer and is generated by applying at least one filter, and/or wherein a first layer and a second layer following the first layer are present, wherein the second layer is generated by applying at least one filter. (Sun, Paragraph [0043], “In some examples, the initial processing module can include a Deep Convolutional Neural Network (Deep CNN). In such examples, the Deep CNN can process the input image through multiple convolutional layers, maximum pooling layers, region of interest pooling layers, and/or fully-connected layers”)
The rationale applied to the rejection of claim 33 has been incorporated herein.
As per Claim 35, Horowitz in view of Sun teaches the method according to claim 33, wherein: a. the convolutional neural network has a decision layer and multiple preceding layers, wherein the decision layer is connected to at least two layers; and/or b. a decision layer of the convolutional neural network has an unsupervised learning algorithm and the decision layer is fully connected to a preceding layer; and/or c. the convolutional neural network has a decision layer and an input layer, wherein the decision layer is connected to the input layer. (Sun, Paragraph [0043], “In some examples, the initial processing module can include a Deep Convolutional Neural Network (Deep CNN). In such examples, the Deep CNN can process the input image through multiple convolutional layers, maximum pooling layers, region of interest pooling layers, and/or fully-connected layers”)
The rationale applied to the rejection of claim 33 has been incorporated herein.
As per Claim 36, Horowitz in view of Sun teaches the method according to claim 32, wherein: a. the artificial neural network has an unsupervised learning algorithm; and/or b. a decision layer of the artificial neural network has the unsupervised learning algorithm. (Sun, Paragraph [0116], “The model may be trained using supervised and/or unsupervised learning. For instance, over time, as the machine learning mechanism receives more training images, the object classifications can be updated based on training image data”)
The rationale applied to the rejection of claim 32 has been incorporated herein.
As per Claim 37, Horowitz in view of Sun teaches the method according to claim 32, wherein a data element of the image signal supplied to a decision layer is evaluated to determine whether it contains a part of the predetermined transport item, wherein the evaluation comprises determining at least one parameter, wherein by using the determined at least one parameter it is determined whether the data element contains a part of the predetermined transport item. (Sun, Paragraph [0045], “ In various examples, the object proposal module may include a Region Proposal Network (RPN), which may be a neural network. In such examples, the RPN can process the convolutional feature map and hypothesize candidate objects and corresponding locations thereof”)
The rationale applied to the rejection of claim 32 has been incorporated herein.
As per Claim 38, Horowitz in view of Sun teaches the method according to claim 37, wherein: a. an unsupervised learning algorithm uses a training result of the artificial neural network in order to determine whether the data element of the image signal contains a part of the predetermined transport item; and/or b. the unsupervised learning algorithm determines that the supplied data element contains a part of the predetermined transport item if the determined at least one parameter is within at least one pre-trained parameter range. (Sun, Paragraph [0045], “ In various examples, the object proposal module may include a Region Proposal Network (RPN), which may be a neural network. In such examples, the RPN can process the convolutional feature map and hypothesize candidate objects and corresponding locations thereof” and Paragraph [0116], “The model may be trained using supervised and/or unsupervised learning. For instance, over time, as the machine learning mechanism receives more training images, the object classifications can be updated based on training image data”)
The rationale applied to the rejection of claim 37 has been incorporated herein.
As per Claim 39, Horowitz in view of Sun teaches the method according to claim 38, wherein: a. the unsupervised learning algorithm is configured such that it outputs a bounding box as an output, which encloses at least a part of the predetermined transport item; and/or b. the unsupervised learning algorithm generates a bounding box when it is determined that a part of the predetermined transport item is arranged in the monitoring area. (Sun, Paragraph [0045], “ In various examples, the object proposal module may include a Region Proposal Network (RPN), which may be a neural network. In such examples, the RPN can process the convolutional feature map and hypothesize candidate objects and corresponding locations thereof. Based on the hypothesis, the RPN can draw proposals in the form of a bounding box around each candidate object in the convolutional feature map. In various examples, the bounding box can be rectangular. However, in other examples, the bounding box can be circular, hexagonal, or other shape”)
The rationale applied to the rejection of claim 38 has been incorporated herein.
As per Claim 40, Horowitz in view of Sun teaches the method according to claim 32, wherein: a. a capture time for generating the image is determined; and/or b. a capture time for generating the image is determined, wherein the capture time is offset by a period of time from a determination time at which the other artificial neural network has determined that at least a part of the object is arranged in the monitoring area; and/or c. a capture time is offset by a period of time from a determination time at which the artificial neural network has determined that the object is arranged in the monitoring area, wherein the period of time is selected such that the entire object is arranged in the monitoring area at the capture time. (Horowitz, Paragraph [0047], “image frames captured by the optical material characterization system 610 may be further tagged with location information (shown at 910) and time and date information (shown at 920) so that collected objects 655 may be correlated to where and when they were collected”)
The rationale applied to the rejection of claim 32 has been incorporated herein.
As per Claim 41, Horowitz in view of Sun teaches the method according to claim 32, wherein: a. a transport item quality is assessed based on the generated image; and/or b. the other neural network has another convolutional neural network; or c. the other neural network has another convolutional neural network, which has fewer layers than a convolutional neural network of the artificial neural network. (Sun, Paragraph [0043])
The rationale applied to the rejection of claim 32 has been incorporated herein.
As per Claim 42, Horowitz in view of Sun teaches the method according to claim 32, wherein a decision layer of the other artificial neural network has another unsupervised learning algorithm, wherein: a. the other unsupervised learning algorithm uses a training result of the other artificial neural network to determine whether a data element of the image signal contains a part of the object; and/or b. the other unsupervised learning algorithm determines another parameter based on the image signal, and the other unsupervised learning algorithm determines whether at least a part of the object is arranged in the monitoring area depending on the other parameter. (Sun, Paragraph [0043] and [0116])
The rationale applied to the rejection of claim 32 has been incorporated herein.
As per Claim 48, Horowitz in view of Sun teaches the method according to claim 32, wherein, for training the other neural network, training images are supplied which contain different objects, and/or which contain objects which are different from the predetermined transport item, and wherein for a training image supplied to a decision layer of the other neural network, at least one other parameter is determined which characterizes whether the training image has at least a part of the predetermined transport item. (Sun, Paragraph [0046], “Block 230 can represent an object classifier module with logic to program processing unit 202 to evaluate the candidate objects proposed by the object proposal module. In various examples, the object classifier module can evaluate each proposal and determine a classification (e.g., a type, a class, a group, a category, etc.) of the candidate object in the proposal. In some examples, the classification of the object can be based on a pre-determined fixed number of object classes. For example, the object classifier can evaluate the object and determine that the object is one of the twenty (20) pre-determined object classes. For another example, the object classifier can evaluate the object and determine that the object is one of the thirty (30) pre-determined object classes. In other examples, the object classifier may receive updated classes of objects periodically”)
The rationale applied to the rejection of claim 32 has been incorporated herein.
As per Claim 49, Claim 49 claims a computing device performing the method as claimed in Claim 32. Therefore the rejection and rationale are analogous to that made in Claim 32.
As per Claim 50, Horowitz in view of Sun teaches a device having an image acquisition device for acquiring an image signal which originates from a monitoring area, and a computing device according to claim 49 which is connected to the image acquisition device in terms of data transmission and to which the acquired image signal is supplied, wherein the image acquisition device captures the image after receiving a capture signal output by the computing device (Horowitz, Figure 1)
The rationale applied to the rejection of claim 49 has been incorporated herein.
As per Claim 51, Horowitz in view of Sun teaches a computer program product comprising instructions which, (Sun, Paragraph [0103]) when the program is executed by a computing device, cause it to carry out the method according to claim 32.
The rationale applied to the rejection of claim 32 has been incorporated herein.
Claims 43-47 are rejected under 35 U.S.C. 103 as being unpatentable over Horowitz et al. US2019/0130560 hereinafter referred to as Horowitz in view of Sun et al. US2017/0206431 hereinafter referred to as Sun as applied to Claim 32 and further in view of Navarrette Michelini et al. US2022/0084166 hereinafter referred to as Navarrette Michelini.
As per Claim 43, Horowitz in view of Sun teaches the method according to claim 32, wherein:
Horowitz in view of Sun does not explicitly teach a. a training of the artificial neural network has a first training phase and a second training phase; and/or b. a training of the artificial neural network has a first training phase and a second training phase, wherein the training of the artificial neural network in the second training phase is carried out using the artificial neural network trained in the first training phase.
Navarrette Michelini teaches a. a training of the artificial neural network has a first training phase and a second training phase; and/or b. a training of the artificial neural network has a first training phase and a second training phase, wherein the training of the artificial neural network in the second training phase is carried out using the artificial neural network trained in the first training phase. (Navarrette Michelini, Paragraph [0206], “Similar to the training at the first phase, in the training at a second phase, based on the trained neural network 100 at the first phase, the trained discriminative network 200 at the first phase is trained at the second phase to improve the discriminative ability of the discriminative network 200 and obtain the trained discriminative network 200 at the second phase; then, based on the trained discriminative network 200 at the second phase, the trained neural network 100 at the first phase is trained at the second phase, so as to improve the image enhancement processing ability of the neural network 100 and obtain the trained neural network 100 at the second phase; and so on”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Navarrette Michelini into Horowitz in view of Sun because by utilizing a training methodology of Navarrette Michelini to train the neural network of Sun will provide proper training of the neural network in order for the neural network to output accurate results.
Therefore it would have been obvious to one of ordinary skill to combine the three references to obtain the invention in Claim 43.
As per Claim 44, Horowitz in view of Sun and Navarrette Michelini teaches the method according to claim 43, wherein in the first training phase: a. the artificial neural network to be trained has another decision layer; and/or b. a decision layer of the artificial neural network to be trained does not have an unsupervised learning algorithm; and/or c. a number of training images supplied to the artificial neural network to be trained in the first training phase is greater than a number of images supplied to the artificial neural network to be trained in the second training phase; and/or d. the images supplied to the artificial neural network to be trained in the first training phase are labeled; and/or e. the images supplied to the artificial neural network to be trained in the first training phase contain the predetermined transport item. (Navarrette Michelini, Paragraph [0206] and Sun, Paragraph [0116])
The rationale applied to the rejection of claim 43 has been incorporated herein.
As per Claim 45, Horowitz in view of Sun and Navarrette Michelini teaches the method according to claim 43, wherein: a. in the second training phase, a decision layer of the artificial neural network to be trained has an unsupervised learning algorithm; and/or b. in the second training phase, training images without labeling are supplied to the artificial neural network; and/or c. in the second training phase, the artificial neural network to be trained is supplied with training images which contain the predetermined transport item and/or training images which do not contain the predetermined transport item. (Navarrette Michelini, Paragraph [0206] and Sun, Paragraph [0116])
The rationale applied to the rejection of claim 43 has been incorporated herein.
As per Claim 46, Horowitz in view of Sun and Navarrette Michelini teaches the method according to claim 43, wherein at least one parameter is determined for a training data element of a training image supplied to a decision layer. (Navarrette Michelini, Paragraph [0206] and Sun, Paragraph [0116])
The rationale applied to the rejection of claim 43 has been incorporated herein.
As per Claim 47, Horowitz in view of Sun and Navarrette Michelini teaches the method according to claim 46, wherein: a. a variance and/or an expected value of image information contained in the training data element of a training image is determined; and/or b. at least one parameter range is determined for the training images supplied in the second training phase, taking into account the at least one determined parameter, in which at least one training data element has a part of the transport object. (Horowitz, Paragraph [0024] and Sun, Paragraph [0043])
The rationale applied to the rejection of claim 46 has been incorporated herein.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MING HON whose telephone number is (571)270-5245. The examiner can normally be reached M-F 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached on 571-270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING Y HON/Primary Examiner, Art Unit 2666
Prosecution Insights