DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
At [0026], “The evaluation appartaus 100 is, for example…” should be written as “The evaluation apparatus 100 is, for example…”
At [0052], layers 220, 230, 240, 250, and 260 are described. However, only R220 associated with layer 220 is defined. Figure 2 references R230, R240, R250, and R260 are not defined in the disclosure.
Appropriate correction is required.
Claim Objections
Claim 1 is objected to because of the following informalities:
In step (b), “…predetermined by the evaluation;;” should be written as “…predetermined by the evaluation;”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 6, and 7 recite the limitation “…a reference evaluation predetermined by the evaluation;” in step (b). There is insufficient antecedent basis for this limitation in the claims, as “the evaluation” is not properly defined before its mention in the claims.
As such, Claims 2-5 are also rejected due to their dependency on Claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7 are rejected under 35 U.S.C. 101.
Step 1:
Claims 1-5 are directed to a method, thus each claim falls under one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Claim 6 is directed to an apparatus, thus the claim falls under one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Claim 7 is directed to a non-transitory computer-readable storage medium, thus the claim falls under one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Regarding Claim 1,
Step 2A:
Based on the claims being determined to be within of the four categories (Step 1), it must be determined if the claims are directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), in this case the claims fall within the judicial exception of an abstract idea.
Step 2A Prong 1:
“(d) calculating a spectral similarity that is a similarity between the reference feature spectrum and the target feature spectrum;” – This limitation recites calculating a similarity value between two other data sets, specifically named “spectrums”, which amounts to a mathematical concept.
“and (e) evaluating the target data using the spectral similarity.” – This limitation recites evaluating data based on a data set, which amounts to a mathematical concept and a mental process.
Step 2A Prong 2:
“(a) inputting a plurality of training sets to a vector neural network machine learning model having a plurality of vector neuron layers…, the training sets including general-purpose training data having a type different from the target data and a label corresponding to the general-purpose training data;” – This limitation recites inputting data, or “training sets,” into an AI model, which amounts to mere data gathering, considered an insignificant extra-solution activity (see MPEP 2106.05(g)).
“(b) after the step (a), inputting reference data having the same type as the target data to the trained machine learning model…, the reference data indicating a reference evaluation predetermined by the evaluation;” – This limitation recites inputting data, or “training sets,” into an AI model, which amounts to mere data gathering, considered an insignificant extra-solution activity (see MPEP 2106.05(g)).
“(c) after the step (a), inputting the target data to be evaluated to the trained machine learning model to acquire a target feature spectrum as the feature spectrum from an output of the specific layer;” – This limitation recites inputting data, or “training sets,” into a machine learning model to retrieve data from a layer, which amounts to mere data gathering, considered an insignificant extra-solution activity (see MPEP 2106.05(g)).
In step (a), “…to train the machine learning model…” – This limitation recites the purpose and use of the data input. Furthermore, the training of the model is recited at a high level of generality using the result of the abstract ideas described above, therefore, it amounts to applying the judicial exception to the field of use for training a machine learning model to perform a task (see MPEP 2106.05(h)).
In step (b), “…to acquire a reference feature spectrum as a feature spectrum from an output of a specific layer of the trained machine learning model…” – This limitation recites obtaining reference data from the machine learning model, which amounts to insignificant extra-solution activity (see MPEP 2106.05(g)), specifically mere data gathering.
Step 2B:
When considered individually or in combination, the additional limitations and elements of Claim 1 are not sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements recited at Claim 1 above amount to applying the judicial exception to a field of use and insignificant extra-solution activity. Moreover, re-evaluation of the additional elements or combination of elements that were considered to be insignificant extra-solution activity at Step 2A: Prong 2 are needed to determine if they are considered well-understood, routine and conventional limitations. The limitations described in steps “(a)”, “(b)”, and “(c)” recite merely receiving data over a network, and retrieving information in memory, and the courts have recognized the following as well-understood, routine, and conventional activity: “Storing and retrieving information in memory” (see MPEP 2106.05(d)(II)(iv)) and “Receiving or transmitting data over a network” (see MPEP 2106.05(d)(II)(i)).
Regarding Claim 2,
Step 2A:
Based on the claims being determined to be within of the four categories (Step 1), it must be determined if the claims are directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), in this case the claims fall within the judicial exception of an abstract idea.
Step 2A Prong 1:
“…in (e), the target data is evaluated according to a classification related to two or more classes…” – This limitation clarifies that the target data is evaluated into a class against multiple others, which amounts to a mental process.
“…the reference evaluation is an evaluation classified into a reference class…” – This limitation clarifies the category of class the reference evaluation belongs, which amounts to a mental process.
“…in (e), the target data is classified into the reference class when the spectral similarity is equal to or larger than a predetermined threshold value…” – This limitation recites how the target data is categorized into the same class as the reference when compared against a threshold value, which amounts to a mathematical concept.
“…and the target data is classified into a class different from the reference class when the spectral similarity is less than the threshold value.” - This limitation recites how the target data is categorized into a different class from the reference when compared against a threshold value, which amounts to a mathematical concept.
Step 2A Prong 2:
This judicial exception, in Claim 2, is not integrated into a practical application because it does not recite any further additional elements.
Step 2B:
Claim 2 does not recite any further additional elements and is not sufficient to amount to significantly more than the judicial exception.
Regarding Claim 3,
Step 2A:
Based on the claims being determined to be within of the four categories (Step 1), it must be determined if the claims are directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), in this case the claims fall within the judicial exception of an abstract idea.
Step 2A Prong 1:
As the limitations in Claim 3 are dependent on the limitations of Claim 1, they recite the same abstract ideas described above. The limitations in Claim 3 do not recite abstract ideas themselves, but they add additional elements to the claim depended upon.
Step 2A Prong 2:
“…the plurality of vector neuron layers include, in order from a side of the target data that is input data, a convolutional vector neuron layer that is an intermediate layer and a classification vector neuron layer that is an output layer…” – This limitation clarifies where each layer, the convolutional and classification vector neuron layers specifically, resides in the model, intermediate and output layers respectively. This amounts to applying the judicial exception to the field of use.
“…and the specific layer is the intermediate layer.” – This limitation clarifies the specific layer to be the intermediate layer within the model, which amounts to an insignificant extra-solution activity. This amounts to applying the judicial exception to the field of use.
Step 2B:
When considered individually or in combination, the additional limitations and elements of Claim 3 are not sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements recited at Claim 3 above amount to applying the judicial exception to a field of use.
Regarding Claim 4,
Step 2A:
Based on the claims being determined to be within of the four categories (Step 1), it must be determined if the claims are directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), in this case the claims fall within the judicial exception of an abstract idea.
Step 2A Prong 1:
“…and the evaluation method further comprises: (f) generating… a plurality of processed reference frame images in which the reference object is extracted, thereby generating the plurality of processed reference frame images arranged in time series as the reference data.” – Examiner interprets this limitation to be directed towards a mathematical concept in light of the specification. (“The data processing unit 113 calculates an absolute value of a difference between each pixel of the reference frame image FMK and an average value of corresponding pixels for each of the plurality of reference frame images FMK1 to FMKN, and generates a set of absolute values of differences in the pixels as the processed reference frame image FMS.” [0061]). It can be seen the processing unit calculates a set of absolute values of differences of reference frame images which generate the reference frame images.
Step 2A Prong 2:
“… each of the general-purpose training data, the reference data, and the target data is a motion image constituted by a plurality of frame images arranged in time series…” – This limitation clarifies the training data, the reference data, and the target data to be a series of frames from a motion image, which amounts to applying the judicial exception to the field of use.
“…using a plurality of reference frame images constituting an original reference motion image acquired by imaging movement of a reference object…” – This limitation clarifies that the image frames described above are calculated by using the frames of a motion picture, which amounts to applying the judicial exception to the field of use.
Step 2B:
When considered individually or in combination, the additional limitations and elements of Claim 4 are not sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements recited at Claim 4 above amount to applying the judicial exception to a field of use.
Regarding Claim 5,
Step 2A:
Based on the claims being determined to be within of the four categories (Step 1), it must be determined if the claims are directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), in this case the claims fall within the judicial exception of an abstract idea.
Step 2A Prong 1:
“…and the evaluation method further comprises: (g) generating… a plurality of processed target frame images in which the evaluation object is extracted, thereby generating the plurality of processed target frame images arranged in time series as the target data.” – Examiner interprets this limitation to be directed towards a mathematical concept in light of the specification. (“The data processing unit 113 executes, by the same method as the image processing of generating the reference data IDS based on the original reference motion image RD, image processing of extracting the moving arm 902 of the robot 900 as the evaluation object from the original target motion image.” [0071]). It can be seen that the target frame images are processed using the same method as the reference frame images, which as stated in the rejection for Claim 4, amounts to calculating.
Step 2A Prong 2:
“… each of the general-purpose training data, the reference data, and the target data is a motion image constituted by a plurality of frame images arranged in time series…” – This limitation clarifies the training data, the reference data, and the target data to be a series of frames from a motion image, which amounts to applying the judicial exception to the field of use.
“…using a plurality of target frame images constituting an original target motion image acquired by imaging movement of an evaluation object…” – This limitation clarifies that the image frames described above are calculated by using the frames of a motion picture, which amounts to applying the judicial exception to the field of use.
Step 2B:
When considered individually or in combination, the additional limitations and elements of Claim 5 are not sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements recited at Claim 5 above amount to applying the judicial exception to a field of use.
Regarding Claim 6 and Claim 7,
The limitations described in the claims are substantially similar to the limitations described in Claim 1 and are directed to the same abstract idea. Thus, they fail to amount to significantly more than the judicial exception and are rejected under the same rationale as Claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 6-7 are rejected under 35 U.S.C. 103 as being unpatentable in view of Kurasawa et al. (US 20210374535 A1 – hereinafter Kurasawa) and further in view of Lee (US 20200242415 A1 – hereinafter Lee).
Regarding Claim 1,
Kurasawa teaches the following:
“(a) inputting a plurality of training sets to a vector neural network machine learning model having a plurality of vector neuron layers to train the machine learning model…” – Kurasawa (“FIG. 5 is a diagram for explaining a setting method of the model 30 of the vector neural network type. The model 30 includes a convolution layer 33, a primary neuron layer 35, a first neuron layer 37, a second neuron layer 38, and a classification neuron layer 39 that is a final layer in this order from an input first data set 12 side.” [0044] & “Next, in step S14, each of the first data elements 12A to 12C of the first data set 12 is sequentially input into the model 30.” [0054]) – Examiner interprets the model described in Kurasawa to be equivalent to the claimed neural network as both are a vector neural network type model. Furthermore, the “plurality of training sets” input into the model are described as “first data elements” in Kurasawa.
“(b) after the step (a), inputting… data having the same type as the target data to the trained machine learning model to acquire a… feature spectrum as a feature spectrum from an output of a specific layer of the trained machine learning model, the… data indicating a… evaluation predetermined by the evaluation;” – Kurasawa (“Specifically, the processor 24 calculates the similarity between the feature spectrum Sp generated from the second intermediate data of the second data elements 62A to 62C and the feature spectrum(.)[sic] Sp generated from the first intermediate data of the first data element 12C of the non-defective product label in each of the first neuron layer 37, the second neuron layer 38, and the classification neuron layer 39. In the present embodiment, the calculation targets of the similarity are the partial regions Rx belonging to the same hierarchy.” [0068]) Examiner interprets the first intermediate data to be the training data sets and the second intermediate data to be the target data. Here, a feature spectrum is generated based on the first intermediate data, specifically output from the intermediate layers, i.e. first and second neuron layers.
“(c) after the step (a), inputting the target data to be evaluated to the trained machine learning model to acquire a target feature spectrum as the feature spectrum from an output of the specific layer;” – Kurasawa (“Specifically, the processor 24 calculates the similarity between the feature spectrum Sp generated from the second intermediate data of the second data elements 62A to 62C and the feature spectrum(.)[sic] Sp generated from the first intermediate data of the first data element 12C of the non-defective product label in each of the first neuron layer 37, the second neuron layer 38, and the classification neuron layer 39. In the present embodiment, the calculation targets of the similarity are the partial regions Rx belonging to the same hierarchy.” [0068]) Examiner interprets the second intermediate data to be equivalent to the target data, where the second intermediate data is compared with the first intermediate data to determine if an image contains a defect. The second intermediate data is calculated to generate a feature spectrum from the first and second neuron layers, i.e. the intermediate layers.
“(d) calculating a spectral similarity that is a similarity between the… feature spectrum and the target feature spectrum;” – Kurasawa (“Specifically, the processor 24 calculates the similarity between the feature spectrum Sp generated from the second intermediate data of the second data elements 62A to 62C and the feature spectrum(.)[sic] Sp generated from the first intermediate data…” [0068]) Examiner interprets the similarity between the second and first feature spectrum, each being the target data and the training data respectively
“and (e) evaluating the target data using the spectral similarity.” – Kurasawa (“The comparison information is information generated when the processor 24 compares the similarity with a predetermined threshold value in step S40. In the data element of the second data set 62, when the similarity is smaller than the predetermined threshold value, it may be interpreted that the similarity with the feature of the known image in the hierarchy is low, and when the similarity is equal to or more than predetermined threshold value, it may be interpreted that the similarity with the feature of the known image in the hierarchy is high. It is discriminated to be the non-defective product.” [0069]) Examiner interprets the similarity to be a similarity “score” based on the two spectrums garnered from the first and second data sets. Using the similarity information, the data is identified to be either defective or non-defective.
Kurasawa does NOT teach the following:
“…the training sets including general-purpose training data having a type different from the target data and a label corresponding to the general-purpose training data;” – Kurasawa does not recite the training data to be general-purpose data, or even a different subject matter from the target data,
“(b) after the step (a), inputting reference data having the same type as the target data to the trained machine learning model to acquire a reference feature spectrum as a feature spectrum from an output of a specific layer of the trained machine learning model, the reference data indicating a reference evaluation predetermined by the evaluation;”” – Kurasawa does not recite using a reference to compare with the target, as it directly compares the target with the training data.
“(d) calculating a spectral similarity that is a similarity between the reference feature spectrum and the target feature spectrum;” – Once more, Kurasawa does not recite using a reference to compare with the target, as it directly compares the target with the training data.
Lee teaches the following:
“…the training sets including general-purpose training data having a type different from the target data and a label corresponding to the general-purpose training data;” – Lee (“Referring back to FIG. 2B, FIG. 2B is a flowchart illustrating another training method of a neural network according to an embodiment of the invention. FIG. 2A and FIG. 2B have the same steps, and a main difference there between is that a third image and a fourth image may be randomly sampled from the first dataset and a second dataset to train the neural network to further recognize a defect degree of a third object. In an embodiment, the label data may correspond to the third object. The third object is different to the first object and the second object...” [0042] & “In summary, the invention provides a supervised training method of a neural network, and the user may tag one of the two images to generate the label data according to the two images. When the neural network is trained, the input parameter may include the label data respectively corresponding to different objects. In this way, the trained neural network may be adapted to objects of different types or models.” [0049]) Examiner interprets the training data to be potentially general-purpose and unrelated to the specific target image, as the model described in Lee refers to object one and object two as part of the trained dataset. However, it can then be used to recognize defects in object three, which is stated to be distinct from the first and second objects and later is clarified to allow the model to compare different objects potentially unrelated to the training data.
“(b) after the step (a), inputting reference data having the same type as the target data to the trained machine learning model to acquire a reference feature spectrum as a feature spectrum from an output of a specific layer of the trained machine learning model, the reference data indicating a reference evaluation predetermined by the evaluation;” – Lee (“The reference image may correspond to the first object, and the target image may correspond to the second object different to the first object, and the classification result of the target image is used for indicating whether an object in the target image has an appearance defect, and the classification result is related to a difference between defect degrees of the reference image and the target image.” [0042] & “It should be tagged that in some embodiments, the aforementioned target image, the reference image 500 and the reference image 600 may respectively correspond to the same or different models or types, which is not limited by the invention.” [0047]) Examiner interprets the reference image to be equivalent to the claimed reference data claimed above. It later clarifies how the reference image and the compared target image can be different or the same type, or subject matter.
“(d) calculating a spectral similarity that is a similarity between the reference feature spectrum and the target feature spectrum;” – Lee (“The reference image may correspond to the first object, and the target image may correspond to the second object different to the first object, and the classification result of the target image is used for indicating whether an object in the target image has an appearance defect, and the classification result is related to a difference between defect degrees of the reference image and the target image.” [0042]) Examiner interprets the reference image to be equivalent to the claimed reference data claimed above. The reference image and the target image are clarified to also be used to calculate a “degree of defect”.
Considering Kurasawa and Lee together:
Lee is analogous art because it is directed towards training a machine learning model to detect defects in an object based on comparing two images given, a target image and a reference image. In consideration of both Kurasawa and Lee together, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize flexible training datasets to create an adaptive machine learning model. Lee (“Since the neural network is generated according to the label data corresponding to different objects (for example, the first object, the second object, and the third object), the neural network may be adapted to different objects (for example, the first object, the second object, and the third object).” [0035]).
In consideration of both Kurasawa and Lee together, it also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a reference image to create dynamic subject matter focus when performing comparisons. Lee (“When the neural network is used for classifying, the user may intuitively adjust the objects for the neural network or the classification criteria of the neural network by means of changing the reference image.” [0048]).
Regarding Claim 2,
The combination of Kurasawa and Lee discloses all of the limitations of Claim 1. The same combination also discloses the limitations of Claim 2.
Kurasawa teaches the following:
“…wherein in (e), the target data is evaluated according to a classification related to two or more classes…” – Kurasawa (“By inputting the second data elements 62A to 62C one by one into the learned model 30, the class discrimination, that is, the label is discriminated. For example, when the second data element 62A indicating one spot or the second data element 62B indicating two spots is input into the model 30, it is discriminated that the product is defective, and when the second data element 62C indicating three spots is input, it is discriminated that the product is non-defective.” [0059] & “In this way, the similarity can be used as a discrimination basis for the class determination of the non-defective product, the defective product, or the like.” [0069]) Examiner interprets the target data, or the second data element to be classified under two possible labels, either defective or non-defective. It is also suggested that the classifications need not be restricted to only defective or non-defective.
“…and in (e), the target data is classified into the… class when the spectral similarity is equal to or larger than a predetermined threshold value…” – Kurasawa (“In the data element of the second data set 62, when the similarity is smaller than the predetermined threshold value, it may be interpreted that the similarity with the feature of the known image in the hierarchy is low, and when the similarity is equal to or more than predetermined threshold value, it may be interpreted that the similarity with the feature of the known image in the hierarchy is high. It is discriminated to be the non-defective product. The predetermined threshold value is, for example, a reference value indicating that the similarity is high or low. In this way, the similarity can be used as a discrimination basis for the class determination of the non-defective product, the defective product, or the like.” [0069]) – Examiner interprets the second data set to be the target data being compared with a similarity threshold. When the second data set is equal to or larger than the threshold value, it is suggested to be non-defective, of which, the training data sets are.
“…and the target data is classified into a class different from the… class when the spectral similarity is less than the threshold value.” – Kurasawa (“In the data element of the second data set 62, when the similarity is smaller than the predetermined threshold value, it may be interpreted that the similarity with the feature of the known image in the hierarchy is low, and when the similarity is equal to or more than predetermined threshold value, it may be interpreted that the similarity with the feature of the known image in the hierarchy is high. It is discriminated to be the non-defective product. The predetermined threshold value is, for example, a reference value indicating that the similarity is high or low. In this way, the similarity can be used as a discrimination basis for the class determination of the non-defective product, the defective product, or the like.” [0069]) – Examiner interprets the second data set to be the target data being compared with a similarity threshold. When the second data set is less than the threshold value, it is suggested to be defective, of which, the training data sets are NOT.
Kurasawa does NOT teach the following:
“…the reference evaluation is an evaluation classified into a reference class…” –Kurasawa does not describe using a reference image during the process, as it compares the target image with the trained datasets directly.
“…and in (e), the target data is classified into the reference class when the spectral similarity is equal to or larger than a predetermined threshold value…” – Kurasawa does not describe using a reference image during the process, as it compares the target image with the trained datasets directly.
“…and the target data is classified into a class different from the reference class when the spectral similarity is less than the threshold value.” – Kurasawa does not describe using a reference image during the process, as it compares the target image with the trained datasets directly.
Lee teaches the following:
“…the reference evaluation is an evaluation classified into a reference class…” – Lee (“For example, when the target object is a display panel of the model A, the user of the classification device may input a reference image corresponding to the display panel of the model A to the neural network to serve as the classification criteria of the neural network.” [0049]) Examiner interprets the reference image to be the deciding factor in classifying the target image as defective or non-defective, thus making it equivalent in function with the reference evaluation which classifies the reference class, i.e. anomaly or no anomaly.
“…and in (e), the target data is classified into the reference class when the spectral similarity is equal to or larger than a predetermined threshold value…” – Lee (“The reference image may serve as criteria for determining whether the target image has the appearance defect. When the defect degree of the appearance of the target image is severer than the defect degree of the appearance of the reference image, the classification device 30 may classify the product corresponding to the target image as a defective product.” [0043]) Examiner interprets the comparison performed is based on the reference image, serving as a classification criteria, compared with the target image to determine if it belongs in the same classification or not, i.e. defective or non-defective.
“…and the target data is classified into a class different from the reference class when the spectral similarity is less than the threshold value.” – Lee (“The reference image may serve as criteria for determining whether the target image has the appearance defect. When the defect degree of the appearance of the target image is severer than the defect degree of the appearance of the reference image, the classification device 30 may classify the product corresponding to the target image as a defective product.” [0043]) Examiner interprets the comparison performed is based on the reference image, serving as a classification criteria, compared with the target image to determine if it belongs in the same classification or not, i.e. defective or non-defective.
Considering Kurasawa and Lee together:
Lee is analogous art because it is directed towards training a machine learning model to detect defects in an object based on comparing two images given, a target image and a reference image. In consideration of both Kurasawa and Lee together, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a reference image as a classification criteria to create dynamic classification criteria when performing comparisons. Lee (“When the neural network is used for classifying, the user may intuitively adjust the objects for the neural network or the classification criteria of the neural network by means of changing the reference image.” [0048).
Regarding Claim 3,
The combination of Kurasawa and Lee discloses all of the limitations of Claim 1. Kurasawa further discloses the limitations of Claim 3.
Kurasawa teaches the following:
“…the plurality of vector neuron layers include, in order from a side of the target data that is input data, a convolutional vector neuron layer that is an intermediate layer and a classification vector neuron layer that is an output layer…” – Kurasawa (“The model 30 includes a convolution layer 33, a primary neuron layer 35, a first neuron layer 37, a second neuron layer 38, and a classification neuron layer 39 that is a final layer in this order from an input first data set 12 side.” [0044]) Examiner interprets the neuron layers making up the model in Kurasawa to be in a similar order as the claimed neuron layers, i.e. in order from an input data side. The first and second neuron layers make up the intermediate layer, performing the same calculations and serving the same purpose as the extra convolutional layers claimed, and the output layer is the final layer, which is the classification layer.
“…and the specific layer is the intermediate layer.” – Kurasawa (“Specifically, the processor 24 calculates each feature spectrum Sp from the first intermediate data and the second intermediate data for each of the partial regions R37, R38, and R39 of each of the first neuron layer 37, the second neuron layer 38, and the classification neuron layer 39.” [0065]) Examiner interprets the specific layer as the layer where the feature spectrum is acquired, as stated in Claim 1. In Kurasawa, the feature spectrum is calculated from the first and second layers which are equivalent to the intermediate layers as described above.
Regarding Claim 6,
Claim 6 is substantially similar to Claim 1, except that it is directed towards an apparatus performing the method of Claim 1. As such, the method performed by the apparatus is rejected under Kurasawa and further in view of Lee for the same rationale as in the rejection for Claim 1.
Regarding the apparatus itself:
“An evaluation apparatus for evaluating target data, the evaluation apparatus comprising:…” – Kurasawa (“According to a third aspect of the present disclosure, an apparatus is provided. The apparatus includes: one or more processors.” [0010]) In light of the specification, examiner interprets the apparatus performing the evaluation method claimed to be equivalent to the apparatus in Kurasawa in structure.
Regarding Claim 7,
Claim 7 is substantially similar to Claim 1, except that it is directed towards a computer-readable storage medium storing the method of Claim 1. As such, the method stored in the medium is rejected under Kurasawa and further in view of Lee for the same rationale as in the rejection for Claim 1.
Regarding the computer-readable storage medium itself:
“A non-transitory computer-readable storage medium storing a program causing a computer to execute an evaluation of target data, the program comprising:…” – Kurasawa (“According to a fifth aspect of the present disclosure, a non-temporary computer-readable medium storing instructions for causing one or more processors to execute…” [0012]) In light of the specification, examiner interprets the computer-readable storage medium performing the evaluation method claimed to be equivalent to the computer-readable medium in Kurasawa in structure, with non-temporary having a similar definition to non-transitory, both meaning permanent.
Claims 4 and 5 are rejected under 35 U.S.C. 103 as being unpatentable in view of Kurasawa and Lee, and further in view of Jones (US 20200125923 A1).
Regarding Claim 4,
The combination of Kurasawa and Lee discloses all of the limitations of Claim 1. Jones further discloses the limitations of Claim 4.
Kurasawa and Lee do NOT teach the following:
“…each of the general-purpose training data, the reference data, and the target data is a motion image constituted by a plurality of frame images arranged in time series…” – Neither Kurasawa nor Lee describe possibly using motion images as training, reference, or target data.
“…and the evaluation method further comprises: (f) generating, using a plurality of reference frame images constituting an original reference motion image acquired by imaging movement of a reference object, a plurality of processed reference frame images in which the reference object is extracted, thereby generating the plurality of processed reference frame images arranged in time series as the reference data.” – Neither Kurasawa nor Lee describe possibly using motion images as reference data.
Jones teaches the following:
“…each of the general-purpose training data, the reference data, and the target data is a motion image constituted by a plurality of frame images arranged in time series…” – Jones (“The image processing system 100 is configured to detect anomalies in a video using a neural network 135 trained to compare two video patches to declare the compared video patches as similar or dissimilar. Using the neural network 135, the imaging system 100 implements an anomaly detector that compares video patches of the input video of a scene to video patches of the training video of the same scene to declare anomalies when a patch of input video is dissimilar to all or corresponding patches in the training video.” [0028]) Examiner interprets the video patches to be equivalent to the motion images claimed. The neural network described in Jones is trained on video patches, making the training data motion images, and compares two video patches, equivalent in function with the reference and target motion images, to detect anomalies. It is also specified that the video patches can make up a plurality of patches corresponding to a full video.
“…and the evaluation method further comprises: (f) generating, using a plurality of reference frame images constituting an original reference motion image acquired by imaging movement of a reference object, a plurality of processed reference frame images in which the reference object is extracted, thereby generating the plurality of processed reference frame images arranged in time series as the reference data.- Jones (“Armed with this understanding, some embodiments train and/or use a neural network not to classify the abnormal vs. normal activity, but to compare video patches of a video of a scene. A video patch includes all of the pixels contained in a spatio-temporal region of a video. In such a manner, instead of providing to a neural network one video patch to classify that video patch as normal or abnormal, the embodiments submit to the neural network 135 two video patches to compare. One video patch is from a database of normal activity and another video patch is from an input video that needs to be classified as normal or abnormal.” [39]) Examiner interprets the video patches to be equivalent to the motion images claimed. The neural network described in Jones is typically comparing the target image with the trained images stored during training. However, it also expresses a variant of the method where the neural network is given two examples, one of a normal case and an unknown case, each the reference and target respectively.
Considering Kurasawa, Lee, and Jones together:
Jones is considered analogous art because it is directed towards using a neural network to determine if a video frame contains an anomaly based on a reference frame. In consideration of both Kurasawa, Lee, and Jones together, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use training, reference, and target data as video or motion image formats to increase computational efficiency. Jones (“It is an object of some embodiments to use direct comparison between the activities in the input video and the activities in the training video for anomaly detection. Such a direct comparison, e.g., the comparison based on Euclidean distance, is computationally efficient and can consider even rare normal motions.” [0010])
Regarding Claim 5,
The combination of Kurasawa and Lee discloses all of the limitations of Claim 1. Jones further discloses the limitations of Claim 5.
Kurasawa and Lee do NOT teach the following:
“…each of the general-purpose training data, the reference data, and the target data is a motion image constituted by a plurality of frame images arranged in time series…” – Neither Kurasawa nor Lee describe possibly using motion images as training, reference, or target data.
“…and the evaluation method further comprises: (g) generating, using a plurality of target frame images constituting an original target motion image acquired by imaging movement of an evaluation object, a plurality of processed target frame images in which the evaluation object is extracted, thereby generating the plurality of processed target frame images arranged in time series as the target data.” – Neither Kurasawa nor Lee describe possibly using motion images as reference data.
Jones teaches the following:
“…each of the general-purpose training data, the reference data, and the target data is a motion image constituted by a plurality of frame images arranged in time series…” – Jones (“The image processing system 100 is configured to detect anomalies in a video using a neural network 135 trained to compare two video patches to declare the compared video patches as similar or dissimilar. Using the neural network 135, the imaging system 100 implements an anomaly detector that compares video patches of the input video of a scene to video patches of the training video of the same scene to declare anomalies when a patch of input video is dissimilar to all or corresponding patches in the training video.” [0028]) Examiner interprets the video patches to be equivalent to the motion images claimed. The neural network described in Jones is trained on video patches, making the training data motion images, and compares two video patches, equivalent in function with the reference and target motion images, to detect anomalies. It is also specified that the video patches can make up a plurality of patches corresponding to a full video.
“…and the evaluation method further comprises: (g) generating, using a plurality of target frame images constituting an original target motion image acquired by imaging movement of an evaluation object, a plurality of processed target frame images in which the evaluation object is extracted, thereby generating the plurality of processed target frame images arranged in time series as the target data.”- Jones (“Armed with this understanding, some embodiments train and/or use a neural network not to classify the abnormal vs. normal activity, but to compare video patches of a video of a scene. A video patch includes all of the pixels contained in a spatio-temporal region of a video. In such a manner, instead of providing to a neural network one video patch to classify that video patch as normal or abnormal, the embodiments submit to the neural network 135 two video patches to compare. One video patch is from a database of normal activity and another video patch is from an input video that needs to be classified as normal or abnormal.” [39]) Examiner interprets the video patches to be equivalent to the motion images claimed. The neural network described in Jones is typically comparing the target image with the trained images stored during training. However, it also expresses a variant of the method where the neural network is given two examples, one of a normal case and an unknown case, each the reference and target respectively.
Considering Kurasawa, Lee, and Jones together:
Jones is considered analogous art because it is directed towards using a neural network to determine if a video frame contains an anomaly based on a reference frame. In consideration of both Kurasawa, Lee, and Jones together, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use training, reference, and target data as video or motion image formats to increase computational efficiency. Jones (“It is an object of some embodiments to use direct comparison between the activities in the input video and the activities in the training video for anomaly detection. Such a direct comparison, e.g., the comparison based on Euclidean distance, is computationally efficient and can consider even rare normal motions.” [0010])
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Wu et al. (US-11210775-B1) – This art is pertinent as it is directed towards using a neural network to detect unusual activity in a video.
Yoon et al. (US-20200320402-A1) – This art is pertinent as it is directed towards a method of using a deep-learning to detect “novel” data within an image.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAMERON CASTANARES whose telephone number is (571)270-5703. The examiner can normally be reached Monday - Thursday (8AM - 6PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAMERON RILEY CASTANARES/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights