DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4/27/2026 has been entered.
Information Disclosure Statement
The information disclosure statement submitted on 1/23/2026 has been considered.
Response to Amendment
Applicant’s Amendment and remarks dated 4/27/2026 have been considered. Clams 2, 4, 18, and 19 are cancelled. Claims 1-3, 5-17, and 20 are pending.
Response to Arguments
On page 8 of Applicant’s 4/27/2026 Amendment and remarks, Applicant asserts that no new matter has been added via the claim amendments.
The examiner acknowledges that at least original claims 2 and 4, together with paras. 0023-0024 and 0039 provide sufficient written description support for the claim amendments.
On page 10 of Applicant’s 4/27/2026 Amendment and remarks, Applicant argues with respect to the rejection of claim 1 under 35 U.S.C. 103:
PNG
media_image1.png
552
650
media_image1.png
Greyscale
The examiner respectfully disagrees. While some language from claims 2 and 4 were incorporated into independent claims 1 and 20, additional claim language requiring the “matching processing is performed between a feature amount extracted from the training data and a feature amount extracted from each of the plurality of pieces of inference sub-data” is a new limitation.
The examiner further notes that the teaching of SUTHAR is not limited to the specific features of that embodiment, but that SUTHAR generally teaches the concept of ranking and listing features according to a score. The present rejections explain in more detail how the combination of WU with CODELLA and SUTHAR, perform a “matching” process as in CODELLA with respect to features of CODELLA, and then rank and list such features according to SUTHAR, such that certain image data from WU is excluded.
On page 11 of Applicant’s 4/27/2026 Amendment and remarks, Applicant argues with respect to the rejection of claim 1 under 35 U.S.C. 103:
PNG
media_image2.png
142
658
media_image2.png
Greyscale
The examiner respectfully submits that the combination of WU, CODELLA, and SUTHAR achieves similar results by prioritizing training data such that less important training data is not used.
On page 12 of Applicant’s 4/27/2026 Amendment and remarks, Applicant makes arguments with respect to the GANTI reference.
The examiner respectfully submits that such arguments are moot because the GANTI reference is no longer used in any rejection in view of the amendments made to independent claims 1 and 20.
On pages 13-14 of Applicant’s 4/27/2026 Amendment and remarks, Applicant makes arguments with respect to claim 17, which has been entirely re-written.
Applicant’s arguments with respect to claim 17 are not persuasive. The combination of WU, MOIN, CODELLA, and SUTHAR teaches this claim, where the combination of WU, MOIN, and CODELLA teaches the “assigning a score based on three-dimensional measurement data” as explained in the detailed rejections.
Claim Objections
Claim 17 is objected to because of the following informalities:
In claim 17, line 8, “a processor configured to execute the program m stored on the memory” should read “a processor configured to execute the program [[m]] stored on the memory”
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-3, 5-17, and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The examiner notes that Applicant’s amendments to the independent claims, which emphasize the matching between training data and inference sub-data, and assigning evaluation scores, necessitates the new rejections under 35 U.S.C. 101.
Regarding Step 1 of the Alice/Mayo framework, Claims 1-3 and 5-17 are directed to a device (an apparatus), and Claim 20 is directed to a method (a process), which each fall within one of the four statutory categories of inventions.
Regarding Claim 1
Step 2A, prong 1 (Is the claim directed to a law of nature, a natural phenomenon or an abstract idea).
Claim 1 recites the following mental processes, that in each case under the broadest reasonable interpretation, covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper but for the recitation of generic computer components (e.g., “processor”, “non-transitory memory”, “trained model”).
perform matching processing between the training data and each of the plurality of pieces of inference sub-data, wherein the matching processing is performed between a feature amount extracted from the training data and a feature amount extracted from each of the plurality of pieces of inference sub-data (under the broadest reasonable interpretation, a human such as a machine learning engineer can perform this recited matching processing between training and inference data based on calculated feature amounts, such as for images of dogs and cats, performing matching processing based on features such as eyes, noses, and whiskers)
assign an evaluation score according to a matching degree to each of the plurality of pieces of inference sub-data (under the broadest reasonable interpretation, a human such as a machine learning engineer can assign an evaluation score based on degrees of matching, such as giving higher scores to higher matches of the features such as eyes, noses and whiskers)
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub-data based on priority depending on the assigned evaluation score (under the broadest reasonable interpretation, a human such as a machine learning engineer can sort and rank the pieces of inference sub-data based on the evaluation score)
Step 2A, prong 2 (Does the claim recite additional elements that integrate the judicial exception into a practical application?).
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements (e.g., “processor”, “non-transitory memory”, “trained model”) which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “a non-transitory memory configured to store a program; a processor configured to execute the program stored on the memory to cause the inference calculation processing device to” limitation, such limitations are recited at a high-level of generality and amount to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional elements of generic processors and memories. These additional elements are recited at a high-level of generality and amount to no more than mere instructions to apply the exception using generic computer components (generic processors and memories). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “acquire the inference data, the trained model, and training data that has been used in machine learning to generate the trained model” limitation, such additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process (see MPEP 2106.05(g)).
Regarding the “divide the acquired inference data into a plurality of pieces of inference sub-data by way of batch processing” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “execute the inference calculation processing for the inference data according to the processing sequence list, based on each of at least one of the plurality of pieces of inference sub-data and the trained model” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of executing training inference data in a particular order. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (executing inference data with a generic model). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Step 2B (Does the claim recite additional elements that amount to significantly more than the judicial exception?)
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements (e.g., “processor”, “non-transitory memory”, “trained model”) are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “a non-transitory memory configured to store a program; a processor configured to execute the program stored on the memory to cause the inference calculation processing device to” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “acquire the inference data, the trained model, and training data that has been used in machine learning to generate the trained model” limitation, as discussed above, the additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “divide the acquired inference data into a plurality of pieces of inference sub-data by way of batch processing” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “execute the inference calculation processing for the inference data according to the processing sequence list, based on each of at least one of the plurality of pieces of inference sub-data and the trained model” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Accordingly, at Step 2B after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Regarding Claim 2
Step 2A, Prong 2
Regarding the “perform the batch processing on the inference data based on the training data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “perform the batch processing on the inference data based on the training data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 5
Step 2A, Prong 2
Regarding the “acquire image data as the inference data” limitation, such additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process (see MPEP 2106.05(g)).
Step 2B
Regarding the “acquire image data as the inference data” limitation, as discussed above, the additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding Claim 6
Step 2A, Prong 2
Regarding the “perform image processing on the image data acquired as the inference data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “perform image processing on the image data acquired as the inference data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 7
Step 2A, Prong 2
Regarding the “perform the batch processing on the inference data based on a result of the image processing” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “perform the batch processing on the inference data based on a result of the image processing” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 8
Step 2A, Prong 1
assign the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the image processing (under the broadest reasonable interpretation, a human an assign an evaluation score to each piece of inference sub-data, such as an image, after it has undergone image processing such as filtering)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 9
Step 2A, Prong 2
Regarding the “acquire voice data as the inference data” limitation, such additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process (see MPEP 2106.05(g)).
Step 2B
Regarding the “acquire voice data as the inference data” limitation, as discussed above, the additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding Claim 10
Step 2A, Prong 1
perform feature analysis on the voice data acquired as the inference data (under the broadest reasonable interpretation, a human can perform such feature analysis, such as listening to voice data and counting the number of pauses)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 11
Step 2A, Prong 2
Regarding the “perform the batch processing on the inference data based on a result of the feature analysis” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “perform the batch processing on the inference data based on a result of the feature analysis” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 12
Step 2A, Prong 1
assign the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the feature analysis. (under the broadest reasonable interpretation, a human an assign an evaluation score to each piece of inference sub-data, such as an image, after it has gone through feature analysis, such as the number of pauses in the audio)
Regarding Claim 13
Step 2A, Prong 2
Regarding the “acquire character data as the inference data” limitation, such additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process (see MPEP 2106.05(g)).
Step 2B
Regarding the “acquire character data as the inference data” limitation, as discussed above, the additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding Claim 14
Step 2A, Prong 1
perform feature analysis on the character data acquired as the inference data (under the broadest reasonable interpretation, a human can perform such feature analysis, such as reading character data and recording the number of “a” characters as a feature)
Regarding Step 2A, Prong 2, the claim does not include any additional elements that integrate the judicial exception into a practical application and regarding Step 2B, there are no additional elements recited that amount to significantly more than the judicial exception.
Regarding Claim 15
Step 2A, Prong 2
Regarding the “perform batch processing on the inference data based on a result of feature analysis” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Step 2B
Regarding the “perform batch processing on the inference data based on a result of feature analysis” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding Claim 16
Step 2A, Prong 1
assign the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the feature analysis. (under the broadest reasonable interpretation, a human an assign an evaluation score to each piece of inference sub-data, such as an image, after it has undergone feature analysis such as the number of “a” characters)
Regarding Claim 17
Step 2A, prong 1 (Is the claim directed to a law of nature, a natural phenomenon or an abstract idea).
Claim 17 recites the following mental processes, that in each case under the broadest reasonable interpretation, covers performance of the limitation in the mind (including an observation, evaluation, judgment, opinion) or with the aid of pencil and paper but for the recitation of generic computer components (e.g., “processor”, “non-transitory memory”, “trained model”).
assign an evaluation score to each of the plurality of pieces of inference sub-data based on the three-dimensional measurement data, (under the broadest reasonable interpretation, a human such as a machine learning engineer can assign an evaluation score based on 3D data)
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub-data based on priority depending on the assigned evaluation score, (under the broadest reasonable interpretation, a human such as a machine learning engineer can sort and rank the pieces of inference sub-data based on the evaluation score)
Step 2A, prong 2 (Does the claim recite additional elements that integrate the judicial exception into a practical application?).
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements (e.g., “processor”, “non-transitory memory”, “trained model”) which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “a non-transitory memory configured to store a program; a processor configured to execute the program stored on the memory to cause the inference calculation processing device to” limitation, such limitations are recited at a high-level of generality and amount to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional elements of generic processors and memories. These additional elements are recited at a high-level of generality and amount to no more than mere instructions to apply the exception using generic computer components (generic processors and memories). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “acquire the inference data, the trained model, and three-dimensional measurement data” limitation, such additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process (see MPEP 2106.05(g)).
Regarding the “divide the acquired inference data into a plurality of pieces of inference sub-data by way of batch processing based on the three-dimensional measurement data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Regarding the “execute the inference calculation processing for the inference data according to the processing sequence list, based on each of at least one of the plurality of pieces of inference sub-data and the trained model” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception. In particular, the claim only recites the additional element of executing training inference data in a particular order. This additional element is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (executing inference data with a generic model). Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Step 2B (Does the claim recite additional elements that amount to significantly more than the judicial exception?)
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements (e.g., “processor”, “non-transitory memory”, “trained model”) are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “a non-transitory memory configured to store a program; a processor configured to execute the program stored on the memory to cause the inference calculation processing device to” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “acquire the inference data, the trained model, and three-dimensional measurement data” limitation, as discussed above, the additional element of a data gathering step is recited at a high level of generality and amounts to extra-solution activity of receiving data, i.e. pre-solution activity of gathering data for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “divide the acquired inference data into a plurality of pieces of inference sub-data by way of batch processing based on the three-dimensional measurement data” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation attempts to cover a solution to an identified problem with no restriction on how the result is accomplished, or provides no description of the mechanism for accomplishing the result. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Regarding the “execute the inference calculation processing for the inference data according to the processing sequence list, based on each of at least one of the plurality of pieces of inference sub-data and the trained model” limitation, such limitation is recited at a high-level of generality and amounts to no more than adding the words “apply it” (or an equivalent) with the judicial exception, because the limitation merely provides instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. Accordingly, this additional element does not add significantly more than the judicial exception. (See MPEP 2106.05(f)).
Accordingly at Step 2B after considering all claim elements individually and as an ordered combination, it is determined that the claims do not integrate the judicial exception into a practical application.
Regarding Claim 20
Step 2A, Prong 1
Claim 20 recites a method that corresponds to the device of claim 1, and therefore the analysis under Step 2A, Prong 1 with respect to claim 1 also applies to this claim 20.
Step 2A, Prong 2
Claim 20 recites a method that corresponds to the device of claim 1, and therefore the analysis under Step 2A, Prong 2 with respect to claim 1 also applies to this claim 20.
Step 2B
Claim 20 recites a method that corresponds to the device of claim 1, and therefore the analysis under Step 2B with respect to claim 1 also applies to this claim 20.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3, 5-8 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over US 20210224999 A1, hereinafter referenced as WU, in view of US 20170185913 A1, hereinafter referenced as CODELLA, and further in view of US 20200134441 A1, hereinafter referenced as SUTHAR.
Regarding Claim 1
WU discloses:
An inference calculation processing device that inputs inference data to a trained model and executes inference calculation processing for the inference data, the inference calculation processing device comprising: (WU, para. 0014: “In a conventional system, images are input into a neural network (e.g., a segmentation network) to output a predicted label for the image.”;
WU, para. 0027: “Image-processing computing system 130 may generate dense predictions such as segmentation to determine the boundaries of things in an image, where a feature is in a spectra, or identifying words in an audio recording. Image-processing computing system 130 can include code for associating input data with the corresponding label, computing alignment results, determining transformations for label data, computing loss scores, and updating the neural network. In some embodiments, image-processing computing system 130 may comprise more than one neural networks (e.g., a neural network and a warping neural network). The warping neural network may be trained to identify a warping or deformation of a predicted label image or a true label image (which can include corresponding patches) that is predicted to align the images, and the neural network can be trained with the warped (predicted or true) image and the other (true or predicted) image to learn how to predict label data based on input images.”;
Examiner’s Note: WU teaches an image-processing computing system 130 (corresponding to recited “inference calculation processing device”) that inputs image data into a neural network, where the neural network executes “inference calculation processing” on the image data (corresponding to “inference data”))
a non-transitory memory configured to store a program; a processor configured to execute the program stored on the memory to cause the inference calculation processing device to: (WU, para. 0005: “Another embodiment includes a non-transitory computer-readable medium storing a plurality of instructions that when executed by one or more processors perform a method comprising...”)
acquire the inference data, the trained model, and training data that has been used in machine learning to generate the trained model; (WU, para. 0022: “Input computing system 110 may generate or receive input data sets. Input computing system 110 may comprise or be in communication with an input sensor 115. Examples of an input sensor 115 may include a camera (e.g., connected to a microscope), a microphone, a spectrometer, or some other sensor capable of recording data.”;
WU, para. 0024: “Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets ....”
WU, para. 0026: “Image-processing computing system 130 can use one or more input data sets and corresponding labels to train a neural network 135. The neural network can include a convolutional neural network and/or dense neural network. Some input data and corresponding labels may be used for validation and/or testing. Some input data (e.g., that is not associated with labels) can be processed by the (e.g., trained) neural network, which can generate predicted labels.”;
WU, para. 0027: “Image-processing computing system 130 can include code for associating input data with the corresponding label, computing alignment results, determining transformations for label data, computing loss scores, and updating the neural network. In some embodiments, image-processing computing system 130 may comprise more than one neural networks (e.g., a neural network and a warping neural network).”
Examiner’s Note: Input computing system 110 acquires input image data divided into training data and validation data (corresponding to recited “training data” and “inference data”) and the image-processing computing system 130 trains a neural network (corresponding to required “acquire... the trained model”); the logic of the input computing system 110 that obtains image data)
divide the acquired inference data into a plurality of pieces of inference sub-data by way of batch processing; and (WU, para. 0024: “ Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets, dividing the input data set into batches, and dividing individual images into patches (e.g., a cropped version of an image).”;
Examiner’s Note: the logic of the input computing system 110 that pre-processes data, including by dividing the input image data set into batches, corresponds to the recited “pre-processing unit”)
execute the inference calculation processing for the inference data ..., based on each of at least one of the plurality of pieces of inference sub-data and the trained model. (WU, para. 0014: “In a conventional system, images are input into a neural network (e.g., a segmentation network) to output a predicted label for the image.”;
WU, para. 0027: “Image-processing computing system 130 may generate dense predictions such as segmentation to determine the boundaries of things in an image, where a feature is in a spectra, or identifying words in an audio recording. Image-processing computing system 130 can include code for associating input data with the corresponding label, computing alignment results, determining transformations for label data, computing loss scores, and updating the neural network. In some embodiments, image-processing computing system 130 may comprise more than one neural networks (e.g., a neural network and a warping neural network). The warping neural network may be trained to identify a warping or deformation of a predicted label image or a true label image (which can include corresponding patches) that is predicted to align the images, and the neural network can be trained with the warped (predicted or true) image and the other (true or predicted) image to learn how to predict label data based on input images.”;
Examiner’s Note: WU teaches that the image-processing computing system 130 inputs image data into a neural network, where the neural network executes “inference calculation processing” on the image data (corresponding to “inference data”) using the neural network itself (corresponding to recited “trained model”))
However, WU fails to explicitly teach:
perform matching processing between the training data and each of the plurality of pieces of inference sub-data, wherein the matching processing is performed between a feature amount extracted from the training data and a feature amount extracted from each of the plurality of pieces of inference sub-data,
assign the evaluation score according to a matching degree to each of the plurality of pieces of inference sub-data
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub-data based on priority depending on the assigned evaluation score,
... according to the processing sequence list
However, in a related field of endeavor (machine learning systems, see para. 0001), CODELLA teaches and makes obvious:
perform matching processing between the training data and each of the plurality of pieces of inference sub-data, wherein the matching processing is performed between a feature amount extracted from the training data and a feature amount extracted from each of the plurality of pieces of inference sub-data, (CODELLA, para. 0022: “To do this, according to one example, the system first examines how each of its individual features responds to training data compared with client test data. This comparison is based on several novel ideas. First, the system quantizes the feature responses into several bins, allowing statistics to be done with integer arithmetic. The bin count and bin ranges do not need to be fixed. For each feature (or component) in the classifier, it looks at the entire training set of data and aggregates into each bin the number of times the feature has attained a value in that bin's range. Therefore, each feature produces a histogram of its response over the training set. It similarly does this with the client data. At this point, each classifier can now be seen as having two sets of histograms: one set comprising histograms, one for each feature, as determined from the training set, and another set comprising histograms, again one for each feature, but as determined from the client set. According to various embodiments, a method determines how these sets of histograms are to be compared. The method adopted, according to various embodiments, is that of the Jensen-Shannon Divergence, which is well defined for all data, does not require assumptions about histogram distributions, and gives results in a limited range (again, from zero to one) that correspond to the mathematical definition of a metric, that is, a distance. Thus, for each feature for each classifier, the method can compare how similar the training set is to the client set, in a way that makes sense to the client: zero means no differences, one means maximal difference.”;
CODELLA, para. 0027: “In some embodiments, the pairwise dimensional comparison provides a predetermined feature (or component) relationship between predetermined components of training data extraction and the test data extraction providing a higher percentage of certainty of a result and less ambiguity.”;
CODELLA, para. 0031: “In another non-limiting example, assume the client test data shows a collection of ambiguous images of a dog that could also be easily misinterpreted by a machine learning system as being a cat or a mouse. The test data extraction of the client's dog images extracts data such as “whiskers”, “furry”, “wet nose”, “floppy ears”, and “hanging tongue”. The training data of the machine learning system product could include this metadata and others including data representative of a cat such as “whiskers”, “furry”, “tail”, “small pointed ears”, “slit pupils” and data representative of a mouse such as “whiskers”, “tail”, “beady eyes” and “pointed ears”.”;
Examiner’s Note: CODELLA teaches comparing training data and test data on a feature-level, where features can be for particular images (corresponding to “pieces of inference sub-data”); the WU-CODELLA combination now modifies the image processing system of WU to use the techniques of CODELLA to determine how similar features are between training data and the features related to pieces of inference sub-data)
assign the evaluation score according to a matching degree to each of the plurality of pieces of inference sub-data (CODELLA, para. 0022: “To do this, according to one example, the system first examines how each of its individual features responds to training data compared with client test data. This comparison is based on several novel ideas. ...The method adopted, according to various embodiments, is that of the Jensen-Shannon Divergence, which is well defined for all data, does not require assumptions about histogram distributions, and gives results in a limited range (again, from zero to one) that correspond to the mathematical definition of a metric, that is, a distance. Thus, for each feature for each classifier, the method can compare how similar the training set is to the client set, in a way that makes sense to the client: zero means no differences, one means maximal difference.”;
Examiner’s Note: CODELLA teaches comparing training data and test data on a feature-level, and assigning a metric between 0 to 1 about the similarity, where 0 = no difference and 1 means maximal difference; the WU-CODELLA combination now modifies the image processing system of WU to use the techniques of CODELLA to determine how similar features are between training data and the features related to pieces of inference sub-data and to assign a score from 0 to 1, where 0 is no difference and 1 is maximal difference)
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA as explained above. As disclosed by CODELLA, one of ordinary skill would have been motivated to do so in order to compare “predetermined components of training data extraction and the test data extraction providing a higher percentage of certainty of a result and less ambiguity.” (para. 0027).
However, WU and CODELLA fail to explicitly teach:
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub-data based on priority depending on the assigned evaluation score,
... according to the processing sequence list
However, in a related field of endeavor (dynamic monitoring of batch data utilizing machine learning models, see para. 0001), SUTHAR teaches and makes obvious:
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub-data based on priority depending on the assigned evaluation score, (SUTHAR, para. 0028: “In one embodiment, the Selection Component 245 utilizes one or more filter methods, which involve applying statistical measures to assign a score for each feature. The features can then be ranked based on this score, and selected or excluded from the data set. In another embodiment, the Selection Component 245 utilizes a wrapper method, which involves preparing, evaluating, and comparing different combinations of features. A predictive model can then be used to evaluate the combinations of features and a score is assigned for model accuracy.”;
Examiner’s Note: SUTHAR teaches ranking data based on a score (corresponding to recited “processing sequence list”); the WU-CODELLA-SUTHAR combination now modifies the image processing system of WU to score the acquired image data of WU according to the similarity measures of CODELLA and then to rank such acquired image data so that the highest-scored data is input into the trained neural network model of WU first)
execute the inference calculation processing for the inference data according to the processing sequence list, based on each of at least one of the plurality of pieces of inference sub-data and the trained model (SUTHAR, para. 0028: “In one embodiment, the Selection Component 245 utilizes one or more filter methods, which involve applying statistical measures to assign a score for each feature. The features can then be ranked based on this score, and selected or excluded from the data set. In another embodiment, the Selection Component 245 utilizes a wrapper method, which involves preparing, evaluating, and comparing different combinations of features. A predictive model can then be used to evaluate the combinations of features and a score is assigned for model accuracy.”;
Examiner’s Note: SUTHAR teaches ranking data based on a score (corresponding to recited “processing sequence list”); the WU-CODELLA-SUTHAR combination now modifies the image processing system of WU to score the acquired image data of WU (using the scoring methods of CODELLA) and then to rank such acquired image data so that the highest-scored data is input into the trained neural network model of WU first according to the ranked listing of SUTHAR)
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA and SUTHAR as explained above. As disclosed by SUTHAR, one of ordinary skill would have been motivated to do so in order to “filter” the data being provided to the predictive model, including selectively excluding data from being input into the model. (para. 0028). One of ordinary skill would further be motivated to do so in order to prioritize data to reduce the computing resources required for machine learning inference at runtime.
Regarding Claim 3
WU, CODELLA, and SUTHAR disclose the device according to claim 1. WU further teaches:
perform the batch processing on the inference data based on the training data. (WU, para. 0024: “ Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets, dividing the input data set into batches, and dividing individual images into patches (e.g., a cropped version of an image).”;
Examiner’s Note: the logic of the input computing system 110 that pre-processes data, including by dividing the input image data set into batches, corresponds to the recited “pre-processing unit”, which batches the image data (corresponding to recited “inference data”) in a manner so that it can be processed by the neural network, which is trained using the “training data” and therefore the batching is performed “based on the training data” because the training data has informed the configuration of the neural network that accepts the batches as input)
Regarding Claim 5
WU, CODELLA, and SUTHAR disclose the device according to claim 1. WU further teaches:
acquire image data as the inference data. (WU, para. 0022: “Input computing system 110 may generate or receive input data sets. Input computing system 110 may comprise or be in communication with an input sensor 115. Examples of an input sensor 115 may include a camera (e.g., connected to a microscope), a microphone, a spectrometer, or some other sensor capable of recording data.”;
WU, para. 0026: “Image-processing computing system 130 can use one or more input data sets and corresponding labels to train a neural network 135. The neural network can include a convolutional neural network and/or dense neural network. Some input data and corresponding labels may be used for validation and/or testing. Some input data (e.g., that is not associated with labels) can be processed by the (e.g., trained) neural network, which can generate predicted labels.”;
WU, para. 0024: “Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets, dividing the input data set into batches, and dividing individual images into patches (e.g., a cropped version of an image). Input computing system 110 can send input data sets to labeling computing system 120 for labeling and to image-processing computing system 130 for processing (e.g., by neural network 635). For example, input computing system 110 may implement a rule that indicates that any image greater than a threshold size (e.g., in terms of dimensions or number of pixels or voxels) is to be divided into patches (e.g., a predefined number of patches, patches of a given size and/or patches having an overlap of a predetermined amount).”)
Regarding Claim 6
WU, CODELLA, and SUTHAR disclose the device according to claim 5. WU further teaches:
perform image processing on the image data acquired as the inference data. (WU, para. 0024: “Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets, dividing the input data set into batches, and dividing individual images into patches (e.g., a cropped version of an image). Input computing system 110 can send input data sets to labeling computing system 120 for labeling and to image-processing computing system 130 for processing (e.g., by neural network 635). For example, input computing system 110 may implement a rule that indicates that any image greater than a threshold size (e.g., in terms of dimensions or number of pixels or voxels) is to be divided into patches (e.g., a predefined number of patches, patches of a given size and/or patches having an overlap of a predetermined amount).”)
Regarding Claim 7
WU, CODELLA, and SUTHAR disclose the device according to claim 6. WU further teaches:
perform the batch processing on the inference data based on a result of the image processing. (WU, para. 0024: “Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets, dividing the input data set into batches, and dividing individual images into patches (e.g., a cropped version of an image). Input computing system 110 can send input data sets to labeling computing system 120 for labeling and to image-processing computing system 130 for processing (e.g., by neural network 635). For example, input computing system 110 may implement a rule that indicates that any image greater than a threshold size (e.g., in terms of dimensions or number of pixels or voxels) is to be divided into patches (e.g., a predefined number of patches, patches of a given size and/or patches having an overlap of a predetermined amount).”; Examiner’s Note: batching is performed after the image data has been normalized, noise has been removed, and data size has been standardized (corresponding to recited “image processing”), and therefore the batching is “based on the result of the image processing” because it is performed on the output of the image processing techniques)
Regarding Claim 8
WU, CODELLA, and SUTHAR discloses the device according to claim 6. However, WU fails to explicitly teach:
assigns the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the image processing
However, in a related field of endeavor (machine learning systems, see para. 0001), CODELLA teaches and makes obvious:
assigns the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the image processing (CODELLA, para. 0022: “To do this, according to one example, the system first examines how each of its individual features responds to training data compared with client test data. This comparison is based on several novel ideas. ...The method adopted, according to various embodiments, is that of the Jensen-Shannon Divergence, which is well defined for all data, does not require assumptions about histogram distributions, and gives results in a limited range (again, from zero to one) that correspond to the mathematical definition of a metric, that is, a distance. Thus, for each feature for each classifier, the method can compare how similar the training set is to the client set, in a way that makes sense to the client: zero means no differences, one means maximal difference.”;
Examiner’s Note: the WU-CODELLA-SUTHAR combination now modifies the image processing system of WU to score the acquired image data of WU using the teachings of CODELLA after image processing results of WU (see para. 0024))
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA and SUTHAR as explained above. As disclosed by CODELLA, one of ordinary skill would have been motivated to do so in order to compare “predetermined components of training data extraction and the test data extraction providing a higher percentage of certainty of a result and less ambiguity.” (para. 0027).
Claim 20 claims a method that corresponds to the device of claim 1 and is therefore rejected for the same reasons explained above with respect to claim 1.
Claims 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over WU in view of CODELLA and SUTHAR, and further in view of US 9824692 B1, hereinafter referenced as KHOURY.
Regarding Claim 9
WU, CODELLA, and SUTHAR disclose the device according to claim 1. However, WU fails to explicitly teach:
acquire voice data as the inference data.
However, in a related field of endeavor (training neural networks in association with a batch processing of input samples, see col. 1, lines 56-67), KHOURY teaches:
acquire voice data as the inference data. (KHOURY, col. 7, lines 4-8: “(24) Furthermore, as shown in FIG. 2B, another feed-forward neural network 242 is used to perform actual speaker recognition based on the recognition speech sample inputted by the user (via input device 10) after training of the DNN is complete.”;
Examiner’s Note: KHOURY discloses a speaker recognition system where a user inputs speech samples (via input device 10, which can be a microphone); the WU-CODELLA-SUTHAR-KHOURY combination now modifies the imaging processing techniques of WU, where input sensors can include microphones for capturing audio (see paras. 0022 and 0027 of WU), to acquire voice data as in KHOURY).
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA, SUTHAR and KHOURY as explained above. As disclosed by KHOURY, one of ordinary skill would have been motivated to do so in order to identify a speaker, for example, in a security-gate embodiment. (col. 6, lines 18-24). One of ordinary skill would further be motivated to do so in applications where, for example, the identity of a person in an image needs to be confirmed, such as in a security setting, or for another example, to confirm the identity of a particular employee doing a particular task).
Regarding Claim 10
WU, CODELLA, SUTHAR, and KHOURY disclose the device according to claim 9. However, WU fails to explicitly teach:
perform feature analysis on the voice data acquired as the inference data.
However, in a related field of endeavor (training neural networks in association with a batch processing of input samples, see col. 1, lines 56-67), KHOURY teaches:
perform feature analysis on the voice data acquired as the inference data (KHOURY, col. 7, lines 49-55: “Such preprocessing may include applying voice activity detection in order to discard a non-speech part of the signal. The preprocessing may also include partitioning the underlying speech signal into a certain number (W) of overlapping windows, and extracting a certain number (F) of features (e.g., Mel filterbank features) from each of the W overlapping windows.”;
Examiner’s Note: KHOURY discloses a speaker recognition system where a user inputs speech samples (via input device 10, which can be a microphone) and pre-processing steps extract features, such as Mel filterbank features; the WU-CODELLA-SUTHAR-KHOURY combination now modifies the imaging processing techniques of WU, where input sensors can include microphones for capturing audio (see paras. 0022 and 0027 of WU), to acquire voice data as in KHOURY, and performs pre-processing to extract features from the voice signal as in KHOURY).
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA, SUTHAR and KHOURY as explained above. As disclosed by KHOURY, one of ordinary skill would have been motivated to do so in order to identify a speaker, for example, in a security-gate embodiment. (col. 6, lines 18-24). One of ordinary skill would further be motivated to do so in applications where, for example, the identity of a person in an image needs to be confirmed, such as in a security setting, or for another example, to confirm the identity of a particular employee doing a particular task).
Regarding Claim 11
WU, CODELLA, SUTHAR, and KHOURY disclose the device according to claim 10. However, WU, CODELLA, and SUTHAR fail to explicitly teach:
perform the batch processing on the inference data based on a result of the feature analysis.
However, in a related field of endeavor (training neural networks in association with a batch processing of input samples, see col. 1, lines 56-67), KHOURY teaches:
perform the batch processing on the inference data based on a result of the feature analysis. (KHOURY, col. 6, lines 26-34: “FIG. 2A illustrates a general structure of a deep neural network (DNN) having a triplet network architecture for use during training, according to exemplary embodiments of the present invention. Also, FIG. 2A illustrates conceptually the use of a batch process in which P audio samples, their corresponding P positive samples, and a cohort set of N negative speech samples are used to train the first, second, and third feed-forward neural networks.”;
KHOURY, col. 7, lines 49-55: “Such preprocessing may include applying voice activity detection in order to discard a non-speech part of the signal. The preprocessing may also include partitioning the underlying speech signal into a certain number (W) of overlapping windows, and extracting a certain number (F) of features (e.g., Mel filterbank features) from each of the W overlapping windows.”;
Examiner’s Note: KHOURY discloses a speaker recognition system where a user inputs speech samples (via input device 10, which can be a microphone) and pre-processing steps extract features, such as Mel filterbank features, prior to using and training the model; the WU-CODELLA-SUTHAR-KHOURY combination now modifies the imaging processing techniques of WU, where input sensors can include microphones for capturing audio (see paras. 0022 and 0027 of WU), to acquire voice data as in KHOURY, and performs pre-processing to extract features from the voice signal as in KHOURY, and then batch processing is performed after acquisition (as in both WU and KHOURY), and therefore the batching is “based on the result of the feature analysis” because it is performed on the output of the feature analysis)).
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA, SUTHAR and KHOURY as explained above. As disclosed by KHOURY, one of ordinary skill would have been motivated to do so in order to identify a speaker, for example, in a security-gate embodiment. (col. 6, lines 18-24). One of ordinary skill would further be motivated to do so in applications where, for example, the identity of a person in an image needs to be confirmed, such as in a security setting, or for another example, to confirm the identity of a particular employee doing a particular task).
Regarding Claim 12
WU, CODELLA, SUTHAR, and KHOURY disclose the device according to claim 10. However, WU fails to explicitly teach:
assign the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the feature analysis
However, in a related field of endeavor (machine learning systems, see para. 0001), CODELLA teaches and makes obvious:
assign the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the feature analysis (CODELLA, para. 0022: “To do this, according to one example, the system first examines how each of its individual features responds to training data compared with client test data. This comparison is based on several novel ideas. First, the system quantizes the feature responses into several bins, allowing statistics to be done with integer arithmetic. The bin count and bin ranges do not need to be fixed. For each feature (or component) in the classifier, it looks at the entire training set of data and aggregates into each bin the number of times the feature has attained a value in that bin's range. Therefore, each feature produces a histogram of its response over the training set. It similarly does this with the client data. At this point, each classifier can now be seen as having two sets of histograms: one set comprising histograms, one for each feature, as determined from the training set, and another set comprising histograms, again one for each feature, but as determined from the client set. According to various embodiments, a method determines how these sets of histograms are to be compared. The method adopted, according to various embodiments, is that of the Jensen-Shannon Divergence, which is well defined for all data, does not require assumptions about histogram distributions, and gives results in a limited range (again, from zero to one) that correspond to the mathematical definition of a metric, that is, a distance. Thus, for each feature for each classifier, the method can compare how similar the training set is to the client set, in a way that makes sense to the client: zero means no differences, one means maximal difference.”;
Examiner’s Note: the WU-CODELLA-SUTHAR-KHOURY combination now modifies the image processing system of WU to score the data as in CODELLA based on the feature analysis of KHOURY)
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA, SUTHAR and KHOURY as explained above. As disclosed by SUTHAR, one of ordinary skill would have been motivated to do so in order to “filter” the data being provided to the predictive model, including selectively excluding data from being input into the model. (para. 0028). One of ordinary skill would further be motivated to do so in order to prioritize data to reduce the computing resources required for machine learning inference at runtime.
Claims 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over WU in view of CODELLA and SUTHAR and further in view of US 20180197087 A1, hereinafter referenced as LUO.
Regarding Claim 13
WU, CODELLA, and SUTHAR disclose the device according to claim 1. However, WU, CODELLA, and SUTHAR fail to explicitly teach:
acquire character data as the inference data.
However, in a related field of endeavor (training machine learning classifiers, see para. 0012), LUO teaches:
Acquire character data as the inference data (LUO, para. 0004: A computing system is described that generates a first classification model for determining a security classification of data items such as electronic files or documents including text and image content. The first classification model can be generated using at least baseline content data or baseline metadata that are extracted from the documents.”
LUO, para. 0062: “Referring again to the example operation executed by server 202, dimension reduction 208 includes selecting the top features from among the multiple relevant extracted text/words, metadata attributes or context factors identified during pre-processing 206.”;
Examiner’s Note: LUO discloses techniques for classifying items in electronic documents based on extracted text (corresponding to recited “character data”); the WU-CODELLA-SUTHAR-LUO combination now modifies the imaging processing techniques of WU, where input sensors can include a camera (see para. 0022 of WU) that can be used to take images of documents, to acquire character data as in LUO (e.g., from optical character recognition)).
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the machine learning teachings of CODELLA, SUTHAR and LUO as explained above. As disclosed by LUO, one of ordinary skill would have been motivated to do so in order to train a device to automatically identify security classifications for documents. (para. 0023). One of ordinary skill would further be motivated to do so in applications where, for example, text needs to be understood, such as extracting words on signs in images to provide additional context for the analyzed images of WU).
Regarding Claim 14
WU, CODELLA, SUTHAR, and LUO disclose the device according to claim 13. However, WU, CODELLA, and SUTHAR fail to explicitly teach:
perform feature analysis on the character data acquired as the inference data.
However, in a related field of endeavor (training machine learning classifiers, see para. 0012), LUO teaches:
perform feature analysis on the character data acquired as the inference data.
LUO, para. 0062: “Referring again to the example operation executed by server 202, dimension reduction 208 includes selecting the top features from among the multiple relevant extracted text/words, metadata attributes or context factors identified during pre-processing 206.”;
LUO, para. 0064: “In some implementations, certain document attributes are identified or selected as top features based on how often a particular attribute is associated with a particular security classification. For example, selected top features can be based on respective feature sets that include top text/word features (i.e., content data) that contribute to certain security classifications, top metadata features that contribute to certain security classifications, and top contextual factors/features that contribute to certain security classifications”;
LUO, para. 0100: “In example operations, the first iteration of model 310 can be used in the offline batch processing mode to generate security labels/classifications for multiple unlabeled documents. While in the offline batch processing mode, server 202 executes feature extraction logic 306 to scan, analyze, and extract one or more features from the unlabeled documents. The extracted features are provided as inputs to the first iteration of model 310 along data path 312.”
Examiner’s Note: LUO discloses techniques for classifying items in electronic documents based on extracted text (corresponding to recited “character data”) and pre-processing steps extract features, such as content, metadata, and contextual features; the WU-CODELLA-SUTHAR-LUO combination now modifies the imaging processing techniques of WU, where input sensors can include a camera (see para. 0022 of WU) that can be used to take images of documents, to acquire character data as in LUO (e.g., from optical character recognition), and to perform pre-processing to extract features from the text data as in LUO).
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA, SUTHAR and LUO as explained above. As disclosed by LUO, one of ordinary skill would have been motivated to do so in order to train a device to automatically identify security classifications for documents. (para. 0023). One of ordinary skill would further be motivated to do so in applications where, for example, text needs to be understood, such as extracting words on signs in images to provide additional context for the analyzed images of WU).
Regarding Claim 15
WU, CODELLA, SUTHAR, and LUO disclose the device according to claim 14. However, WU, CODELLA, and SUTHAR fail to explicitly teach:
perform batch processing on the inference data based on a result of feature analysis.
However, in a related field of endeavor (training machine learning classifiers, see para. 0012), LUO teaches:
perform batch processing on the inference data based on a result of feature analysis. (LUO, para. 0062: “Referring again to the example operation executed by server 202, dimension reduction 208 includes selecting the top features from among the multiple relevant extracted text/words, metadata attributes or context factors identified during pre-processing 206.”;
LUO, para. 0100: “In example operations, the first iteration of model 310 can be used in the offline batch processing mode to generate security labels/classifications for multiple unlabeled documents. While in the offline batch processing mode, server 202 executes feature extraction logic 306 to scan, analyze, and extract one or more features from the unlabeled documents. The extracted features are provided as inputs to the first iteration of model 310 along data path 312.”
Examiner’s Note: LUO discloses techniques for classifying items in electronic documents based on extracted text (corresponding to recited “character data”) and pre-processing steps extract features, such as content, metadata, and contextual features prior to using the model; the WU-CODELLA-SUTHAR-LUO combination now modifies the imaging processing techniques of WU, where input sensors can include a camera (see para. 0022 of WU) that can be used to take images of documents, to acquire character data as in LUO (e.g., from optical character recognition), and to perform pre-processing to extract features from the text data as in LUO, and then batch processing is performed after acquisition (as in both WU and LUO), and therefore the batching is “based on the result of the feature analysis” because it is performed on the output of the feature analysis)).
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA, SUTHAR and LUO as explained above. As disclosed by LUO, one of ordinary skill would have been motivated to do so in order to train a device to automatically identify security classifications for documents. (para. 0023). One of ordinary skill would further be motivated to do so in applications where, for example, text needs to be understood, such as extracting words on signs in images to provide additional context for the analyzed images of WU).
Regarding Claim 16
WU, CODELLA, SUTHAR, and LUO disclose the device according to claim 14. However, WU fails to explicitly teach:
assign the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the feature analysis
However, in a related field of endeavor (machine learning systems, see para. 0001), CODELLA teaches and makes obvious:
assign the evaluation score to each of the plurality of pieces of inference sub-data based on a result of the feature analysis (CODELLA, para. 0022: “To do this, according to one example, the system first examines how each of its individual features responds to training data compared with client test data. This comparison is based on several novel ideas. First, the system quantizes the feature responses into several bins, allowing statistics to be done with integer arithmetic. The bin count and bin ranges do not need to be fixed. For each feature (or component) in the classifier, it looks at the entire training set of data and aggregates into each bin the number of times the feature has attained a value in that bin's range. Therefore, each feature produces a histogram of its response over the training set. It similarly does this with the client data. At this point, each classifier can now be seen as having two sets of histograms: one set comprising histograms, one for each feature, as determined from the training set, and another set comprising histograms, again one for each feature, but as determined from the client set. According to various embodiments, a method determines how these sets of histograms are to be compared. The method adopted, according to various embodiments, is that of the Jensen-Shannon Divergence, which is well defined for all data, does not require assumptions about histogram distributions, and gives results in a limited range (again, from zero to one) that correspond to the mathematical definition of a metric, that is, a distance. Thus, for each feature for each classifier, the method can compare how similar the training set is to the client set, in a way that makes sense to the client: zero means no differences, one means maximal difference.”;
Examiner’s Note: SUTHAR teaches ranking data based on a score; the CODELLA-WU-SUTHAR-LUO combination now modifies the image processing system of WU to score the data as in CODELLA based on the feature analysis of LUO)
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of CODELLA, SUTHAR and LUO as explained above. As disclosed by SUTHAR, one of ordinary skill would have been motivated to do so in order to “filter” the data being provided to the predictive model, including selectively excluding data from being input into the model. (para. 0028). One of ordinary skill would further be motivated to do so in order to prioritize data to reduce the computing resources required for machine learning inference at runtime.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over WU in view of US 20200320685 A1, hereinafter referenced as MOIN, and further in view of CODELLA and SUTHAR.
Regarding Claim 17
WU discloses:
An inference calculation processing device that inputs inference data to a trained model and executes inference calculation processing for the inference data, the inference calculation processing device comprising: (WU, para. 0014: “In a conventional system, images are input into a neural network (e.g., a segmentation network) to output a predicted label for the image.”;
WU, para. 0027: “Image-processing computing system 130 may generate dense predictions such as segmentation to determine the boundaries of things in an image, where a feature is in a spectra, or identifying words in an audio recording. Image-processing computing system 130 can include code for associating input data with the corresponding label, computing alignment results, determining transformations for label data, computing loss scores, and updating the neural network. In some embodiments, image-processing computing system 130 may comprise more than one neural networks (e.g., a neural network and a warping neural network). The warping neural network may be trained to identify a warping or deformation of a predicted label image or a true label image (which can include corresponding patches) that is predicted to align the images, and the neural network can be trained with the warped (predicted or true) image and the other (true or predicted) image to learn how to predict label data based on input images.”;
Examiner’s Note: WU teaches an image-processing computing system 130 (corresponding to recited “inference calculation processing device”) that inputs image data into a neural network, where the neural network executes “inference calculation processing” on the image data (corresponding to “inference data”))
a non-transitory memory configured to store a program; a processor configured to execute the program stored on the memory to cause the inference calculation processing device to: (WU, para. 0005: “Another embodiment includes a non-transitory computer-readable medium storing a plurality of instructions that when executed by one or more processors perform a method comprising...”)
acquire the inference data, the trained model, ...; (WU, para. 0022: “Input computing system 110 may generate or receive input data sets. Input computing system 110 may comprise or be in communication with an input sensor 115. Examples of an input sensor 115 may include a camera (e.g., connected to a microscope), a microphone, a spectrometer, or some other sensor capable of recording data.”;
WU, para. 0024: “Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets ....”
WU, para. 0026: “Image-processing computing system 130 can use one or more input data sets and corresponding labels to train a neural network 135. The neural network can include a convolutional neural network and/or dense neural network. Some input data and corresponding labels may be used for validation and/or testing. Some input data (e.g., that is not associated with labels) can be processed by the (e.g., trained) neural network, which can generate predicted labels.”;
WU, para. 0027: “Image-processing computing system 130 can include code for associating input data with the corresponding label, computing alignment results, determining transformations for label data, computing loss scores, and updating the neural network. In some embodiments, image-processing computing system 130 may comprise more than one neural networks (e.g., a neural network and a warping neural network).”
Examiner’s Note: Input computing system 110 acquires input image data divided into training data and validation data (where validation data corresponds to recited “inference data”) and the image-processing computing system 130 trains a neural network (corresponding to required “acquire... the trained model”); the logic of the input computing system 110 that obtains image data)
divide the acquired inference data into a plurality of pieces of inference sub-data by way of batch processing ... ; (WU, para. 0024: “ Input computing system 110 may also pre-process the data by, for example, normalizing data, removing noise, and standardizing data size. Pre-processing may also include diving the input data set into training, testing, and validation sets, dividing the input data set into batches, and dividing individual images into patches (e.g., a cropped version of an image).”;
Examiner’s Note: the logic of the input computing system 110 that pre-processes data, including by dividing the input image data set into batches, corresponds to the recited “pre-processing unit”)
execute the inference calculation processing for the inference data ... , based on each of at least one of the plurality of pieces of inference sub-data and the trained model. (WU, para. 0014: “In a conventional system, images are input into a neural network (e.g., a segmentation network) to output a predicted label for the image.”;
WU, para. 0027: “Image-processing computing system 130 may generate dense predictions such as segmentation to determine the boundaries of things in an image, where a feature is in a spectra, or identifying words in an audio recording. Image-processing computing system 130 can include code for associating input data with the corresponding label, computing alignment results, determining transformations for label data, computing loss scores, and updating the neural network. In some embodiments, image-processing computing system 130 may comprise more than one neural networks (e.g., a neural network and a warping neural network). The warping neural network may be trained to identify a warping or deformation of a predicted label image or a true label image (which can include corresponding patches) that is predicted to align the images, and the neural network can be trained with the warped (predicted or true) image and the other (true or predicted) image to learn how to predict label data based on input images.”;
Examiner’s Note: WU teaches that the image-processing computing system 130 inputs image data into a neural network, where the neural network executes “inference calculation processing” on the image data (corresponding to “inference data”) using the neural network itself (corresponding to recited “trained model”))
However, WU fails to explicitly teach:
... and three-dimensional measurement data
... based on the three-dimensional measurement data
assign an evaluation score to each of the plurality of pieces of inference sub-data based on the three-dimensional measurement data,
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub-data based on priority depending on the assigned evaluation score,
... according to the processing sequence list
However, in a related field of endeavor (training neural networks see para. 0002), MOIN teaches:
acquire ... three-dimensional measurement data. (MOIN, para. 0029: “In a further aspect, the invention relates to a method for training a deep neural network system to process 3D image data of a dento-maxillofacial structure. The method may include a computer receiving training data, the training data including: 3D input data, preferably 3D cone beam CT (CBCT) image data, the 3D input data defining one or more voxel representations of one or more dento-maxillofacial structures respectively, a voxel being associated with a radiation intensity value, the voxels of a voxel representation defining an image volume;”
MOIN, para. 0031: “the computer using a pre-processing algorithm to determine 3D positional feature information of the dento-maxillofacial structure, the 3D positional feature information defining for each voxel in the voxel representation information about the position of the voxel relative to the position of a dental reference object, e.g. a jaw, a dental arch and/or one or more teeth, in the image volume”;
Examiner’s Note: MOIN discloses a dental system that uses 3D positional input data to train a neural network to classify types of teeth, using medical imaging technologies to generate 3d image data (see para. 0006); the WU-MOIN combination now modifies the imaging processing techniques of WU to acquire 3d imaging and positional data as in MOIN).
divide the acquired inferenced data ... based on the three-dimensional measurement data (MOIN, para. 0029: “In a further aspect, the invention relates to a method for training a deep neural network system to process 3D image data of a dento-maxillofacial structure. The method may include a computer receiving training data, the training data including: 3D input data, preferably 3D cone beam CT (CBCT) image data, the 3D input data defining one or more voxel representations of one or more dento-maxillofacial structures respectively, a voxel being associated with a radiation intensity value, the voxels of a voxel representation defining an image volume;”
MOIN, para. 0031: “the computer using a pre-processing algorithm to determine 3D positional feature information of the dento-maxillofacial structure, the 3D positional feature information defining for each voxel in the voxel representation information about the position of the voxel relative to the position of a dental reference object, e.g. a jaw, a dental arch and/or one or more teeth, in the image volume”;
Examiner’s Note: MOIN discloses a dental system that uses 3D positional input data to train a neural network to classify types of teeth, using medical imaging technologies to generate 3d image data (see para. 0006); the WU-MOIN combination now modifies the imaging processing techniques of WU to acquire 3d imaging and positional data as in MOIN and then perform division and batch processing on such 3d imaging data as in WU).
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of MOIN as explained above. As disclosed by MOIN, one of ordinary skill would have been motivated to do so in order to train a neural network classifier to classify different parts of the mouth, such as jaw, teeth, and/or nerves, when creating a 3d model of a patient’s mouth. (para. 0047). One of ordinary skill would further be motivated to do so in applications where, for example, 3d positioning information is required, such as facial recognition.
However, WU and MOIN fail to explicitly teach:
assign an evaluation score to each of the plurality of pieces of inference sub-data based on the three-dimensional measurement data,
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub- data based on priority depending on the assigned evaluation score,
... according to the processing sequence list
However, in a related field of endeavor (machine learning systems, see para. 0001), CODELLA teaches and makes obvious:
assign an evaluation score to each of the plurality of pieces of inference sub-data based on the three-dimensional measurement data, (CODELLA, para. 0022: “To do this, according to one example, the system first examines how each of its individual features responds to training data compared with client test data. This comparison is based on several novel ideas. ...The method adopted, according to various embodiments, is that of the Jensen-Shannon Divergence, which is well defined for all data, does not require assumptions about histogram distributions, and gives results in a limited range (again, from zero to one) that correspond to the mathematical definition of a metric, that is, a distance. Thus, for each feature for each classifier, the method can compare how similar the training set is to the client set, in a way that makes sense to the client: zero means no differences, one means maximal difference.”;
Examiner’s Note: CODELLA teaches comparing training data and test data on a feature-level, and assigning a metric between 0 to 1 about the similarity, where 0 = no difference and 1 means maximal difference; the WU-MOIN-CODELLA combination now modifies the image processing system of WU to use the techniques of CODELLA to determine how similar features are between training data and the features related to pieces of inference sub-data and to assign a score from 0 to 1, where 0 is no difference and 1 is maximal difference)
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of MOIN and CODELLA as explained above. As disclosed by CODELLA, one of ordinary skill would have been motivated to do so in order to compare “predetermined components of training data extraction and the test data extraction providing a higher percentage of certainty of a result and less ambiguity.” (para. 0027).
However, WU, MOIN, and CODELLA, fail to explicitly teach:
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub- data based on priority depending on the assigned evaluation score,
... according to the processing sequence list
However, in a related field of endeavor (dynamic monitoring of batch data utilizing machine learning models, see para. 0001), SUTHAR teaches and makes obvious:
generate a processing sequence list indicating an optimized inference calculation processing sequence of the plurality of pieces of inference sub-data based on priority depending on the assigned evaluation score, (SUTHAR, para. 0028: “In one embodiment, the Selection Component 245 utilizes one or more filter methods, which involve applying statistical measures to assign a score for each feature. The features can then be ranked based on this score, and selected or excluded from the data set. In another embodiment, the Selection Component 245 utilizes a wrapper method, which involves preparing, evaluating, and comparing different combinations of features. A predictive model can then be used to evaluate the combinations of features and a score is assigned for model accuracy.”;
Examiner’s Note: SUTHAR teaches ranking data based on a score (corresponding to recited “processing sequence list”); the WU-MOIN-CODELLA-SUTHAR combination now modifies the image processing system of WU to score the acquired image data of WU according to the similarity measures of CODELLA and then to rank such acquired image data so that the highest-scored data is input into the trained neural network model of WU first)
execute the inference calculation processing for the inference data according to the processing sequence list, based on each of at least one of the plurality of pieces of inference sub-data and the trained model (SUTHAR, para. 0028: “In one embodiment, the Selection Component 245 utilizes one or more filter methods, which involve applying statistical measures to assign a score for each feature. The features can then be ranked based on this score, and selected or excluded from the data set. In another embodiment, the Selection Component 245 utilizes a wrapper method, which involves preparing, evaluating, and comparing different combinations of features. A predictive model can then be used to evaluate the combinations of features and a score is assigned for model accuracy.”;
Examiner’s Note: SUTHAR teaches ranking data based on a score (corresponding to recited “processing sequence list”); the WU-MOIN-CODELLA-SUTHAR combination now modifies the image processing system of WU to score the acquired image data of WU (using the scoring methods of CODELLA) and then to rank such acquired image data so that the highest-scored data is input into the trained neural network model of WU first according to the ranked listing of SUTHAR)
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of MOIN, CODELLA and SUTHAR as explained above. As disclosed by SUTHAR, one of ordinary skill would have been motivated to do so in order to “filter” the data being provided to the predictive model, including selectively excluding data from being input into the model. (para. 0028). One of ordinary skill would further be motivated to do so in order to prioritize data to reduce the computing resources required for machine learning inference at runtime.
Before the effective filing date of the present application, one of ordinary skill would have been motivated to combine the machine learning system of WU with the teachings of MOIN, CODELLA and SUTHAR as explained above. As disclosed by SUTHAR, one of ordinary skill would have been motivated to do so in order to “filter” the data being provided to the predictive model, including selectively excluding data from being input into the model. (para. 0028). One of ordinary skill would further be motivated to do so in order to prioritize data to reduce the computing resources required for machine learning inference at runtime.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20190391122 A1 (Imamura). “An evaluation in the time domain will be conducted in accordance with the analysis mode (FIG. 2) in which the sample is identified by comparing the features obtained from the test data with the features obtained from the training data for the respective channels.” (para. 0047).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL C LEE whose telephone number is (571)272-4933. The examiner can normally be reached M-F 12:00 pm - 8:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL C. LEE/Examiner, Art Unit 2128
Prosecution Insights