DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/27/2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to amendments and remarks filed on 07/31/2026, which have been entered. In the current amendments, claims 1-3, 5, 8, 10, 13, and 17-22 are amended. Claims 1-22 are pending and have been examined.
In response to amendments and remarks filed on 07/31/2026, the claim objections, and 35 U.S.C. 112(b) rejections made in the previous office action are withdrawn.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Objections
Claims 1-22 are objected to because of the following informalities:
In claim 1, lines 19-21, “a response map of the input data generated, by a model that responds to a specific color of the input data” should read “a response map of the input data generated by a model that responds to a specific color of the input data”
In claim 13, lines 13-15, “a response map of the input data generated, by a model that responds to a specific color of the input data” should read “a response map of the input data generated by a model that responds to a specific color of the input data”
In claim 17, lines 22-24, “a response map of the input data generated, by a model that responds to a specific color of the input data” should read “a response map of the input data generated by a model that responds to a specific color of the input data”
In claim 18, lines 15-17, “a response map of the input data generated, by a model that responds to a specific color of the input data” should read “a response map of the input data generated by a model that responds to a specific color of the input data”
In claim 19, lines 17-19, “a response map of the input data generated, by a model that responds to a specific color of the input data” should read “a response map of the input data generated by a model that responds to a specific color of the input data”
In claim 20, lines 16-18, “a response map of the input data generated, by a model that responds to a specific color of the input data” should read “a response map of the input data generated by a model that responds to a specific color of the input data”
Dependent claims 2-12 and 14-16 are objected based on being directly or indirectly dependent on objected claim 1.
Dependent claims 21 and 22 are objected based on being directly or indirectly dependent on objected claim 13.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6, and 13-22 are rejected under 35 U.S.C. 103 as being unpatentable over Hachiya et al. (US 2017/0206437 A1) in view of Li et al. (US 2021/0334664 A1) in view of Kiraly et al. (US 2018/0240233 A1) and further in view of Lin et al. (US 2018/0260698 A1).
Regarding Claim 1,
Hachiya et al. teaches an information processing apparatus (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches a recognition training (information processing) apparatus) operable to
train a machine learning model that has a hierarchical structure configured by a plurality of hierarchical layers and that is used for recognizing a recognition target in input data (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches the apparatus training a recognizer to recognize a recognition target in inputted data. Fig. 1; [0051]-[0052]: "The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target … The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN (R-CNN) … can be applied to the recognizer … Processing by the recognition training unit 13 when the R-CNN is used as the recognizer is specifically described here" teaches that the recognizer that is trained is a CNN (hierarchical machine learning model with hierarchical layers)), the apparatus comprising:
at least one processor; and at least one memory having stored thereon instructions which, when executed by the at least one processor, cause the information processing apparatus (Fig. 1; [0127]: "Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) … The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like" teaches the embodied apparatus comprising a processor and a memory storing instructions for execution by the processor to perform the disclosed functions) at least to:
Hachiya et al. does not appear to explicitly teach obtain input data and data indicating a ground truth of an output from the machine learning model regarding the input data; and train the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model, and wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Li et al. teaches obtain input data and data indicating a ground truth of an output from the machine learning model regarding the input data (Fig. 2; [0028]: "the input module 108 is configured to receive a new domain dataset 120 for use in training the domain-specific model 118 to generate the domain-agnostic model 106. The new domain dataset 120 includes new domain data 122 and ground truth data 124, which corresponds to ideal or optimal outputs to be produced by the model when processing the new domain data 122 according to an auxiliary task. In this manner, the new domain dataset 120 is useable by the domain adaptation system 104 to compute a loss function for generating the domain-agnostic model 106 based on the domain-specific model 118" teaches an input module for receiving new domain data (input data) and ground truth data corresponding to ideal outputs (data indicating a ground truth of an output from the machine learning model) from the model when processing new domain data (input data)); and
train the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data (Fig. 2; Fig. 3; [0060]: "the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on the loss (error) between a feature probability distribution 204 for the new domain data generated from the feature networks (intermediate layers) in the domain-specific model (output in an intermediate layer of the machine learning model with respect to the input data) and ground truth data corresponding to the new domain (specific domain) input data (data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data)),
wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model (Fig. 2; Fig. 3; [0057]-[0060]: "To generate the local context vectors 316, the domain-specific model 118 implements a local domain classifier 322, which is representative of a fully-convolutional neural network configured to output a domain prediction map 324 having a same size (e.g., width and height) as the input data 302. In some implementations the local domain classifier 322 is trained using local loss 326. The local loss 326 is representative of any suitable type of loss algorithm for aligning low-level features, such as a least-squares loss. The local context vector 316 is extracted from a middle layer of the local domain classifier 322 … To generate the global context vectors 318, the domain-specific model 118 implements a global domain classifier 328, which is representative of a fully-convolutional neural network configured to predict a domain useable to describe the input data 302 … The global loss 330 is representative of a loss function that influences the global domain classifier 328 to ignore easy-to-classify examples (e.g., instances of the input data 302 being of a same domain as training data used to generate the domain-specific model 118) and focus on difficult-to-classify examples (e.g., instances of the input data 302 being of a different domain as training data used to generate the domain-specific model 118). In this manner, the global loss 330 may be representative of a variety of different known loss functions, such as a cross-entropy loss function, a focal loss function, and the like … By presuming any classified bounding box 320 output by the domain-specific model 118 to be unreliable when the input data 302 is representative of new domain data 122, the domain adaptation system 104 is configured to leverage information generated by the domain-specific model 118 to determine the loss function 206 for use in generating the domain-agnostic model 106 … the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on a local loss 326 (first loss) between the output of a local feature network (intermediate layer) of the model and the ground truth data corresponding to the new domain (specific domain) input data (specific domain ground truth data) and a global loss 330 (second loss) between the output of the global domain classifier of the model (output of a final layer of the machine learning model) and the ground truth data).
Hachiya et al. and Li et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate obtain input data and data indicating a ground truth of an output from the machine learning model regarding the input data; and train the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model as taught by Li et al. to the disclosed invention of Hachiya et al.
One of ordinary skill in the art would have been motivated to make this modification to "enable generation of a trained model configured to handle diverse input data without requiring the size and scope of training data otherwise necessitated by conventional approaches, thereby reducing an amount of computational and network resources used in training a model" (Li et al. [0024]).
Hachiya et al. in view of Li et al. does not appear to explicitly teach wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Kiraly et al. teaches wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on the generated classifications in a slice (response map) and the ground truth data. [0028]-[0029]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice ... the prostate tumor detection and classification results can be output by overlaying the generated benign tumor response maps and malignant tumor response maps on structural MRI images, such as T2-weighted MRI images. For example, 2D response maps generated for benign and malignant tumors can be overlaid on corresponding slices of the T2-weighted MRI image of the patient and displayed on a display device. In a possible embodiment, the generated benign tumor response maps and the generated malignant tumor response maps can be overlaid on slices of the T2-weighted MRI image using different colors to represent detected benign and malignant tumors in order to provide an easily distinguishable indication of the classification of each detected tumor" teaches that the generated classifications in a slice are response maps that are corresponding to a specific domain/classification).
Hachiya et al., Li et al., and Kiraly et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).
Hachiya et al. in view of Li et al. and further in view of Kiraly et al. does not appear to explicitly teach a response map of the input data generated, by a model that responds to a specific color of the input data.
However, Lin et al. teaches a response map of the input data generated, by a model that responds to a specific color of the input data ([0065]: "given a trained CNN, the CNN can process an input image to identify feature maps describing image features in locations in the image. The feature maps are created by processing the image with a convolutional neural network (CNN) which is trained to extract the image features (e.g., data describing objects) based on pixel values (e.g., color, grayscale value) of pixels within locations of the image. The CNN produces feature maps from the extracted image features" teaches the feature maps (response maps) being generated by a CNN (by a model) that responds to specific colors of the input image (input data)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a response map of the input data generated, by a model that responds to a specific color of the input data as taught by Lin et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Kiraly et al.
One of ordinary skill in the art would have been motivated to make this modification "to produce feature maps that better represent the features of the image, thus improving accuracy" (Lin et al. [0026]).
Regarding Claim 6,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
In addition, Hachiya et al. further teaches wherein the instructions further cause the information processing apparatus to: recognize a recognition target in input data for verification using the machine learning model (Fig. 1; [0050]-[0051]: "Further, the recognition target selection unit 12 outputs an assembly of combinations of the extracted recognition target ID and semantic relevance together with the input domain ID and the ontology information to the recognition training unit 13 … The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target. Specifically, in response that the domain ID, the ontology information, and the assembly of combinations of the recognition target ID and the semantic relevance are input from the recognition target selection unit 12, the recognition training unit 13 retains the recognition target ID matching with the input recognition target ID. The recognition training unit 13 reads a line in which the data type information is “training” from the moving image data storage unit M2. The read line includes the recognition target name information, the recognition target ID, and the moving image datum. The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN … can be applied to the recognizer" teaches a recognition target selection unit 12 and recognition training unit 13 to recognize a recognition target in input data for verification using the recognizer, which is a CNN (machine learning model)); and
obtain information indicating the specific domain for which a recognition result needs to be improved in the input data for verification (Fig. 8; [0075]-[0079]: "A recognition training system 1a according to the present exemplary embodiment is described using a case in which the pre-trained recognizer is subjected to fine-tuning as an example. … the recognition training apparatus 10a adaptively trains the recognizer based on the operation information expressing feedback from the user to the recognition target visualized information. … Returning to the description of FIG. 8, the recognition training apparatus 10a is an apparatus for performing fine-tuning on the recognizer with respect to the specific domain. … The recognition target update unit 15 updates the recognition target based on the operation information indicating an operation by a person to the recognition target visualized information displayed on the display unit DS of the terminal apparatus 100. Specifically, the recognition target update unit 15 detects that the domain ID, the operation information “execution of fine-tuning”, and the assembly of the recognition target IDs are input from the terminal apparatus 100" teaches a recognition target update unit 15 that obtains operation information expressing feedback (e.g. needing improvement) for the recognition target visualized information (recognition result) to perform fine-tuning on the recognizer (machine learning model) with respect to the specific domain), and
wherein additional training for the machine learning model is performed in accordance with the information indicating the specific domain (Fig. 8; [0079]-[0086]: "The recognition target update unit 15 updates the recognition target based on the operation information indicating an operation by a person to the recognition target visualized information displayed on the display unit DS of the terminal apparatus 100. Specifically, the recognition target update unit 15 detects that the domain ID, the operation information “execution of fine-tuning”, and the assembly of the recognition target IDs are input from the terminal apparatus 100. In response to the input, the recognition target update unit 15 reads the parameter of the recognizer associated with the domain ID, the assembly of the recognition target IDs, and the assembly of the semantic relevance information pieces from the recognizer storage unit M3. Further, the recognition target update unit 15 updates the assembly of the read recognition target IDs and the read parameter of the recognizer based on the assembly of the input recognition target IDs. Specifically, the recognition target update unit 15 replaces the assembly of the read recognition target IDs with the assembly of the input recognition target IDs. Further, the recognition target update unit 15 updates the read parameter of the recognizer based on the input assembly of the recognition target IDs. As a parameter update method, there are two methods described below. … The recognition target update unit 15 adjusts a training related parameter of the recognizer used by the recognition training unit 13a based on the assembly of the read recognition target IDs and the assembly of the input recognition target IDs. As an adjustment method of the training related parameter, there are, for example, two methods described below. … Further, the recognition target update unit 15 outputs the input domain ID, the assembly of the updated recognition target ID, the updated recognizer parameter, the adjusted training related parameter, and the assembly of the read semantic relevance to the recognition training unit 13a. … The recognition training unit 13a performs fine-tuning on the recognizer" teaches recognition training unit 13a (learning unit) performs fine-tuning (additional training) for the recognizer (machine learning model) based on the specific domain information from the recognition target update unit 15).
Regarding Claim 13,
Hachiya et al. teaches an information processing apparatus (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches a recognition training (information processing) apparatus), comprising:
at least one processor; and at least one memory having stored thereon instructions which, when executed by the at least one processor, cause the information processing apparatus (Fig. 1; [0127]: "Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) … The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like" teaches the embodied apparatus comprising a processor and a memory storing instructions for execution by the processor to perform the disclosed functions) at least to:
recognize a recognition target in the input data using a machine learning model having a hierarchical structure configured by a plurality of layers (Fig. 1; [0050]-[0051]: "Further, the recognition target selection unit 12 outputs an assembly of combinations of the extracted recognition target ID and semantic relevance together with the input domain ID and the ontology information to the recognition training unit 13 … The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target. Specifically, in response that the domain ID, the ontology information, and the assembly of combinations of the recognition target ID and the semantic relevance are input from the recognition target selection unit 12, the recognition training unit 13 retains the recognition target ID matching with the input recognition target ID. The recognition training unit 13 reads a line in which the data type information is “training” from the moving image data storage unit M2. The read line includes the recognition target name information, the recognition target ID, and the moving image datum. The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN … can be applied to the recognizer" teaches a recognition target selection unit 12 and recognition training unit 13 to recognize a recognition target in input data for verification using the recognizer, which is a CNN (hierarchical machine learning model with hierarchical layers)).
Hachiya et al. does not appear to explicitly teach obtain input data; and wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model, and wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Li et al. teaches obtain input data (Fig. 2; [0028]: "the input module 108 is configured to receive a new domain dataset 120 for use in training the domain-specific model 118 to generate the domain-agnostic model 106. The new domain dataset 120 includes new domain data 122 and ground truth data 124, which corresponds to ideal or optimal outputs to be produced by the model when processing the new domain data 122 according to an auxiliary task. In this manner, the new domain dataset 120 is useable by the domain adaptation system 104 to compute a loss function for generating the domain-agnostic model 106 based on the domain-specific model 118" teaches an input module for receiving new domain data (input data)); and
wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model (Fig. 2; Fig. 3; [0057]-[0060]: "To generate the local context vectors 316, the domain-specific model 118 implements a local domain classifier 322, which is representative of a fully-convolutional neural network configured to output a domain prediction map 324 having a same size (e.g., width and height) as the input data 302. In some implementations the local domain classifier 322 is trained using local loss 326. The local loss 326 is representative of any suitable type of loss algorithm for aligning low-level features, such as a least-squares loss. The local context vector 316 is extracted from a middle layer of the local domain classifier 322 … To generate the global context vectors 318, the domain-specific model 118 implements a global domain classifier 328, which is representative of a fully-convolutional neural network configured to predict a domain useable to describe the input data 302 … The global loss 330 is representative of a loss function that influences the global domain classifier 328 to ignore easy-to-classify examples (e.g., instances of the input data 302 being of a same domain as training data used to generate the domain-specific model 118) and focus on difficult-to-classify examples (e.g., instances of the input data 302 being of a different domain as training data used to generate the domain-specific model 118). In this manner, the global loss 330 may be representative of a variety of different known loss functions, such as a cross-entropy loss function, a focal loss function, and the like … By presuming any classified bounding box 320 output by the domain-specific model 118 to be unreliable when the input data 302 is representative of new domain data 122, the domain adaptation system 104 is configured to leverage information generated by the domain-specific model 118 to determine the loss function 206 for use in generating the domain-agnostic model 106 … the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on a local loss 326 (first loss) between the output of a local feature network (intermediate layer) of the model and the ground truth data corresponding to the new domain (specific domain) input data (specific domain ground truth data) and a global loss 330 (second loss) between the output of the global domain classifier of the model (output of a final layer of the machine learning model) and the ground truth data).
Hachiya et al. and Li et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate obtain input data; and wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model as taught by Li et al. to the disclosed invention of Hachiya et al.
One of ordinary skill in the art would have been motivated to make this modification to "enable generation of a trained model configured to handle diverse input data without requiring the size and scope of training data otherwise necessitated by conventional approaches, thereby reducing an amount of computational and network resources used in training a model" (Li et al. [0024]).
Hachiya et al. in view of Li et al. does not appear to explicitly teach wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Kiraly et al. teaches wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on the generated classifications in a slice (response map) and the ground truth data. [0028]-[0029]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice ... the prostate tumor detection and classification results can be output by overlaying the generated benign tumor response maps and malignant tumor response maps on structural MRI images, such as T2-weighted MRI images. For example, 2D response maps generated for benign and malignant tumors can be overlaid on corresponding slices of the T2-weighted MRI image of the patient and displayed on a display device. In a possible embodiment, the generated benign tumor response maps and the generated malignant tumor response maps can be overlaid on slices of the T2-weighted MRI image using different colors to represent detected benign and malignant tumors in order to provide an easily distinguishable indication of the classification of each detected tumor" teaches that the generated classifications in a slice are response maps that are corresponding to a specific domain/classification).
Hachiya et al., Li et al., and Kiraly et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).
Hachiya et al. in view of Li et al. and further in view of Kiraly et al. does not appear to explicitly teach a response map of the input data generated, by a model that responds to a specific color of the input data.
However, Lin et al. teaches a response map of the input data generated, by a model that responds to a specific color of the input data ([0065]: "given a trained CNN, the CNN can process an input image to identify feature maps describing image features in locations in the image. The feature maps are created by processing the image with a convolutional neural network (CNN) which is trained to extract the image features (e.g., data describing objects) based on pixel values (e.g., color, grayscale value) of pixels within locations of the image. The CNN produces feature maps from the extracted image features" teaches the feature maps (response maps) being generated by a CNN (by a model) that responds to specific colors of the input image (input data)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a response map of the input data generated, by a model that responds to a specific color of the input data as taught by Lin et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Kiraly et al.
One of ordinary skill in the art would have been motivated to make this modification "to produce feature maps that better represent the features of the image, thus improving accuracy" (Lin et al. [0026]).
Regarding Claim 14,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
In addition, Li et al. further teaches wherein the specific domain is a portion having a specific color, a portion having a specific spatial frequency, or a portion of a subject of a specific class ([0002]: "conventional approaches to training machine learning models leverage a large amount of labeled data constrained to a specific domain (e.g., large amounts of images of birds with labels identifying different characteristics and/or bird types) to train a model for a specific objective (e.g., bird detection and classification)" teaches that a specific domain is a portion of a subject of a specific class (e.g. characteristics of birds)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain is a portion having a specific color, a portion having a specific spatial frequency, or a portion of a subject of a specific class as taught by Li et al. to the disclosed invention of Hachiya et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "enable generation of a trained model configured to handle diverse input data without requiring the size and scope of training data otherwise necessitated by conventional approaches, thereby reducing an amount of computational and network resources used in training a model" (Li et al. [0024]).
Regarding Claim 15,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
In addition, Hachiya et al. further teaches wherein the specific domain is a case for which it is necessary to perform recognition at a higher accuracy (Fig. 1; [0074]: "The recognition training unit of the recognition training apparatus generates the importance information of the recognition target selected based on the semantic relevance and performs the pre-training on the selected recognition target by weighting based on the importance information. Accordingly, the pre-training of the accuracy of the recognizer can be performed on the recognition target required by more users in the specific domain" teaches that recognition accuracy of the recognizer (machine learning model) is trained based on weighting the importance information for the recognition target in the specific domain (i.e. the specific domain recognition target needs a higher accuracy)).
Regarding Claim 16,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
In addition, Hachiya et al. further teaches wherein the machine learning model classifies a sub-region in input data into a category, detects a recognition target that is present in input data, or classifies input data (Fig. 1; [0050]-[0051]: "Further, the recognition target selection unit 12 outputs an assembly of combinations of the extracted recognition target ID and semantic relevance together with the input domain ID and the ontology information to the recognition training unit 13 … The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target. Specifically, in response that the domain ID, the ontology information, and the assembly of combinations of the recognition target ID and the semantic relevance are input from the recognition target selection unit 12, the recognition training unit 13 retains the recognition target ID matching with the input recognition target ID. The recognition training unit 13 reads a line in which the data type information is “training” from the moving image data storage unit M2. The read line includes the recognition target name information, the recognition target ID, and the moving image datum. The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN … can be applied to the recognizer" teaches that the recognizer, which is a CNN (machine learning model), recognizes (detects) a recognition target in input data).
Regarding Claim 17,
Hachiya et al. teaches an information processing apparatus (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches a recognition training (information processing) apparatus) operable to
train a machine learning model that has a hierarchical structure configured by a plurality of hierarchical layers and that is used for recognizing a recognition target in input data (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches the apparatus training a recognizer to recognize a recognition target in inputted data. Fig. 1; [0051]-[0052]: "The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target … The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN (R-CNN) … can be applied to the recognizer … Processing by the recognition training unit 13 when the R-CNN is used as the recognizer is specifically described here" teaches that the recognizer that is trained is a CNN (hierarchical machine learning model with hierarchical layers)), the apparatus comprising:
at least one processor; and at least one memory having stored thereon instructions which, when executed by the at least one processor, cause the information processing apparatus (Fig. 1; [0127]: "Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) … The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like" teaches the embodied apparatus comprising a processor and a memory storing instructions for execution by the processor to perform the disclosed functions) at least to:
recognize the recognition target in input data using a machine learning model having a hierarchical structure configured by a plurality of layers (Fig. 1; [0050]-[0051]: "Further, the recognition target selection unit 12 outputs an assembly of combinations of the extracted recognition target ID and semantic relevance together with the input domain ID and the ontology information to the recognition training unit 13 … The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target. Specifically, in response that the domain ID, the ontology information, and the assembly of combinations of the recognition target ID and the semantic relevance are input from the recognition target selection unit 12, the recognition training unit 13 retains the recognition target ID matching with the input recognition target ID. The recognition training unit 13 reads a line in which the data type information is “training” from the moving image data storage unit M2. The read line includes the recognition target name information, the recognition target ID, and the moving image datum. The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN … can be applied to the recognizer" teaches a recognition target selection unit 12 and recognition training unit 13 to recognize a recognition target in input data for verification using the recognizer, which is a CNN (hierarchical machine learning model with hierarchical layers));
present a result of the recognition target with respect to input data for verification (Fig. 1; [0058]-[0059]: "The recognition target visualization unit 14 displays recognition target information expressing the selected recognition target by superimposing on the ontology information. The recognition target visualization unit 14 calculates the recognition accuracy of the recognizer with respect to each recognition target selected by the recognition target selection unit 12 from evaluation data and generates the recognition target visualized information. … In other words, the recognition target visualization unit 14 calculates accuracy with respect to each recognition target. … The recognition target visualization unit 14 generates the recognition target visualized information visually expressing the recognition target of the recognizer based on the assembly of the input recognition target IDs and the ontology information. … The recognition target visualization unit 14 may superimpose the calculated recognition accuracy of each recognition target on the recognition target visualized information together with the concept information having the recognition target ID matching with the input recognition target ID as the recognition target information. The recognition target visualization unit 14 outputs the generated recognition target visualized information to the terminal apparatus 100" teaches a recognition target visualization unit 14 that displays (presents) recognition target visualized information representing the results from the recognition unit (e.g. recognition accuracy for recognition target) for the input data);
obtain information indicating a specific domain for which a recognition result needs to be improved in the input data for verification (Fig. 8; [0075]-[0079]: "A recognition training system 1a according to the present exemplary embodiment is described using a case in which the pre-trained recognizer is subjected to fine-tuning as an example. … the recognition training apparatus 10a adaptively trains the recognizer based on the operation information expressing feedback from the user to the recognition target visualized information. … Returning to the description of FIG. 8, the recognition training apparatus 10a is an apparatus for performing fine-tuning on the recognizer with respect to the specific domain. … The recognition target update unit 15 updates the recognition target based on the operation information indicating an operation by a person to the recognition target visualized information displayed on the display unit DS of the terminal apparatus 100. Specifically, the recognition target update unit 15 detects that the domain ID, the operation information “execution of fine-tuning”, and the assembly of the recognition target IDs are input from the terminal apparatus 100" teaches a recognition target update unit 15 that obtains operation information expressing feedback (e.g. needing improvement) for the recognition target visualized information (recognition result) to perform fine-tuning on the recognizer (machine learning model) with respect to the specific domain).
Hachiya et al. does not appear to explicitly teach perform training so as to optimize the machine learning model using data indicating a ground truth of an output from the machine learning model with respect to input data for training extracted regarding the specific domain, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model, and wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Li et al. teaches perform training so as to optimize the machine learning model using data indicating a ground truth of an output from the machine learning model with respect to input data for training extracted regarding the specific domain (Fig. 2; Fig. 3; [0060]: "the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on the loss (error) between a feature probability distribution 204 for the new domain data and ground truth data corresponding to the new domain (specific domain) input data (data indicating a ground truth of an output from the machine learning model regarding a specific domain of the input data)),
wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model (Fig. 2; Fig. 3; [0057]-[0060]: "To generate the local context vectors 316, the domain-specific model 118 implements a local domain classifier 322, which is representative of a fully-convolutional neural network configured to output a domain prediction map 324 having a same size (e.g., width and height) as the input data 302. In some implementations the local domain classifier 322 is trained using local loss 326. The local loss 326 is representative of any suitable type of loss algorithm for aligning low-level features, such as a least-squares loss. The local context vector 316 is extracted from a middle layer of the local domain classifier 322 … To generate the global context vectors 318, the domain-specific model 118 implements a global domain classifier 328, which is representative of a fully-convolutional neural network configured to predict a domain useable to describe the input data 302 … The global loss 330 is representative of a loss function that influences the global domain classifier 328 to ignore easy-to-classify examples (e.g., instances of the input data 302 being of a same domain as training data used to generate the domain-specific model 118) and focus on difficult-to-classify examples (e.g., instances of the input data 302 being of a different domain as training data used to generate the domain-specific model 118). In this manner, the global loss 330 may be representative of a variety of different known loss functions, such as a cross-entropy loss function, a focal loss function, and the like … By presuming any classified bounding box 320 output by the domain-specific model 118 to be unreliable when the input data 302 is representative of new domain data 122, the domain adaptation system 104 is configured to leverage information generated by the domain-specific model 118 to determine the loss function 206 for use in generating the domain-agnostic model 106 … the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on a local loss 326 (first loss) between the output of a local feature network (intermediate layer) of the model and the ground truth data corresponding to the new domain (specific domain) input data (specific domain ground truth data) and a global loss 330 (second loss) between the output of the global domain classifier of the model (output of a final layer of the machine learning model) and the ground truth data).
Hachiya et al. and Li et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate perform training so as to optimize the machine learning model using data indicating a ground truth of an output from the machine learning model with respect to input data for training extracted regarding the specific domain, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model as taught by Li et al. to the disclosed invention of Hachiya et al.
One of ordinary skill in the art would have been motivated to make this modification to "enable generation of a trained model configured to handle diverse input data without requiring the size and scope of training data otherwise necessitated by conventional approaches, thereby reducing an amount of computational and network resources used in training a model" (Li et al. [0024]).
Hachiya et al. in view of Li et al. does not appear to explicitly teach wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Kiraly et al. teaches wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on the generated classifications in a slice (response map) and the ground truth data. [0028]-[0029]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice ... the prostate tumor detection and classification results can be output by overlaying the generated benign tumor response maps and malignant tumor response maps on structural MRI images, such as T2-weighted MRI images. For example, 2D response maps generated for benign and malignant tumors can be overlaid on corresponding slices of the T2-weighted MRI image of the patient and displayed on a display device. In a possible embodiment, the generated benign tumor response maps and the generated malignant tumor response maps can be overlaid on slices of the T2-weighted MRI image using different colors to represent detected benign and malignant tumors in order to provide an easily distinguishable indication of the classification of each detected tumor" teaches that the generated classifications in a slice are response maps that are corresponding to a specific domain/classification).
Hachiya et al., Li et al., and Kiraly et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).
Hachiya et al. in view of Li et al. and further in view of Kiraly et al. does not appear to explicitly teach a response map of the input data generated, by a model that responds to a specific color of the input data.
However, Lin et al. teaches a response map of the input data generated, by a model that responds to a specific color of the input data ([0065]: "given a trained CNN, the CNN can process an input image to identify feature maps describing image features in locations in the image. The feature maps are created by processing the image with a convolutional neural network (CNN) which is trained to extract the image features (e.g., data describing objects) based on pixel values (e.g., color, grayscale value) of pixels within locations of the image. The CNN produces feature maps from the extracted image features" teaches the feature maps (response maps) being generated by a CNN (by a model) that responds to specific colors of the input image (input data)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a response map of the input data generated, by a model that responds to a specific color of the input data as taught by Lin et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Kiraly et al.
One of ordinary skill in the art would have been motivated to make this modification "to produce feature maps that better represent the features of the image, thus improving accuracy" (Lin et al. [0026]).
Regarding Claim 18,
Hachiya et al. teaches an information processing method performed by an information processing apparatus (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches a recognition training (information processing) apparatus. [0127]: "Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus" teaches that the embodied apparatus may be used for implementing a method) operable to
train a machine learning model that has a hierarchical structure configured by a plurality of hierarchical layers and that is used for recognizing a recognition target in input data (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches the apparatus training a recognizer to recognize a recognition target in inputted data. Fig. 1; [0051]-[0052]: "The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target … The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN (R-CNN) … can be applied to the recognizer … Processing by the recognition training unit 13 when the R-CNN is used as the recognizer is specifically described here" teaches that the recognizer that is trained is a CNN (hierarchical machine learning model with hierarchical layers)).
Hachiya et al. does not appear to explicitly teach obtaining input data and data indicating a ground truth of an output from the machine learning model regarding the input data; and training the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model, and wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Li et al. teaches obtaining input data and data indicating a ground truth of an output from the machine learning model regarding the input data (Fig. 2; [0028]: "the input module 108 is configured to receive a new domain dataset 120 for use in training the domain-specific model 118 to generate the domain-agnostic model 106. The new domain dataset 120 includes new domain data 122 and ground truth data 124, which corresponds to ideal or optimal outputs to be produced by the model when processing the new domain data 122 according to an auxiliary task. In this manner, the new domain dataset 120 is useable by the domain adaptation system 104 to compute a loss function for generating the domain-agnostic model 106 based on the domain-specific model 118" teaches an input module (obtaining unit) for receiving new domain data (input data) and ground truth data corresponding to ideal outputs (data indicating a ground truth of an output from the machine learning model) from the model when processing new domain data (input data)); and
training the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data (Fig. 2; Fig. 3; [0060]: "the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module (learning unit) to train the domain-agnostic model (machine learning model) based on the loss (error) between a feature probability distribution 204 for the new domain data generated from the feature networks (intermediate layers) in the domain-specific model (output in an intermediate layer of the machine learning model with respect to the input data) and ground truth data corresponding to the new domain (specific domain) input data (data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data)),
wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model (Fig. 2; Fig. 3; [0057]-[0060]: "To generate the local context vectors 316, the domain-specific model 118 implements a local domain classifier 322, which is representative of a fully-convolutional neural network configured to output a domain prediction map 324 having a same size (e.g., width and height) as the input data 302. In some implementations the local domain classifier 322 is trained using local loss 326. The local loss 326 is representative of any suitable type of loss algorithm for aligning low-level features, such as a least-squares loss. The local context vector 316 is extracted from a middle layer of the local domain classifier 322 … To generate the global context vectors 318, the domain-specific model 118 implements a global domain classifier 328, which is representative of a fully-convolutional neural network configured to predict a domain useable to describe the input data 302 … The global loss 330 is representative of a loss function that influences the global domain classifier 328 to ignore easy-to-classify examples (e.g., instances of the input data 302 being of a same domain as training data used to generate the domain-specific model 118) and focus on difficult-to-classify examples (e.g., instances of the input data 302 being of a different domain as training data used to generate the domain-specific model 118). In this manner, the global loss 330 may be representative of a variety of different known loss functions, such as a cross-entropy loss function, a focal loss function, and the like … By presuming any classified bounding box 320 output by the domain-specific model 118 to be unreliable when the input data 302 is representative of new domain data 122, the domain adaptation system 104 is configured to leverage information generated by the domain-specific model 118 to determine the loss function 206 for use in generating the domain-agnostic model 106 … the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on a local loss 326 (first loss) between the output of a local feature network (intermediate layer) of the model and the ground truth data corresponding to the new domain (specific domain) input data (specific domain ground truth data) and a global loss 330 (second loss) between the output of the global domain classifier of the model (output of a final layer of the machine learning model) and the ground truth data).
Hachiya et al. and Li et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate obtaining input data and data indicating a ground truth of an output from the machine learning model regarding the input data; and training the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model as taught by Li et al. to the disclosed invention of Hachiya et al.
One of ordinary skill in the art would have been motivated to make this modification to "enable generation of a trained model configured to handle diverse input data without requiring the size and scope of training data otherwise necessitated by conventional approaches, thereby reducing an amount of computational and network resources used in training a model" (Li et al. [0024]).
Hachiya et al. in view of Li et al. does not appear to explicitly teach wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Kiraly et al. teaches wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on the generated classifications in a slice (response map) and the ground truth data. [0028]-[0029]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice ... the prostate tumor detection and classification results can be output by overlaying the generated benign tumor response maps and malignant tumor response maps on structural MRI images, such as T2-weighted MRI images. For example, 2D response maps generated for benign and malignant tumors can be overlaid on corresponding slices of the T2-weighted MRI image of the patient and displayed on a display device. In a possible embodiment, the generated benign tumor response maps and the generated malignant tumor response maps can be overlaid on slices of the T2-weighted MRI image using different colors to represent detected benign and malignant tumors in order to provide an easily distinguishable indication of the classification of each detected tumor" teaches that the generated classifications in a slice are response maps that are corresponding to a specific domain/classification).
Hachiya et al., Li et al., and Kiraly et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).
Hachiya et al. in view of Li et al. and further in view of Kiraly et al. does not appear to explicitly teach a response map of the input data generated, by a model that responds to a specific color of the input data.
However, Lin et al. teaches a response map of the input data generated, by a model that responds to a specific color of the input data ([0065]: "given a trained CNN, the CNN can process an input image to identify feature maps describing image features in locations in the image. The feature maps are created by processing the image with a convolutional neural network (CNN) which is trained to extract the image features (e.g., data describing objects) based on pixel values (e.g., color, grayscale value) of pixels within locations of the image. The CNN produces feature maps from the extracted image features" teaches the feature maps (response maps) being generated by a CNN (by a model) that responds to specific colors of the input image (input data)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a response map of the input data generated, by a model that responds to a specific color of the input data as taught by Lin et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Kiraly et al.
One of ordinary skill in the art would have been motivated to make this modification "to produce feature maps that better represent the features of the image, thus improving accuracy" (Lin et al. [0026]).
Regarding Claim 19,
Hachiya et al. teaches an information processing method performed by an information processing apparatus (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches a recognition training (information processing) apparatus. [0127]: "Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus" teaches that the embodied apparatus may be used for implementing a method) operable to
train a machine learning model that has a hierarchical structure configured by a plurality of hierarchical layers and that is used for recognizing a recognition target in input data (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches the apparatus training a recognizer to recognize a recognition target in inputted data. Fig. 1; [0051]-[0052]: "The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target … The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN (R-CNN) … can be applied to the recognizer … Processing by the recognition training unit 13 when the R-CNN is used as the recognizer is specifically described here" teaches that the recognizer that is trained is a CNN (hierarchical machine learning model with hierarchical layers)), the method comprising:
recognizing the recognition target in input data using a machine learning model having a hierarchical structure configured by a plurality of layers (Fig. 1; [0050]-[0051]: "Further, the recognition target selection unit 12 outputs an assembly of combinations of the extracted recognition target ID and semantic relevance together with the input domain ID and the ontology information to the recognition training unit 13 … The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target. Specifically, in response that the domain ID, the ontology information, and the assembly of combinations of the recognition target ID and the semantic relevance are input from the recognition target selection unit 12, the recognition training unit 13 retains the recognition target ID matching with the input recognition target ID. The recognition training unit 13 reads a line in which the data type information is “training” from the moving image data storage unit M2. The read line includes the recognition target name information, the recognition target ID, and the moving image datum. The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN … can be applied to the recognizer" teaches a recognition target selection unit 12 and recognition training unit 13 (recognition unit) to recognize a recognition target in input data for verification using the recognizer, which is a CNN (hierarchical machine learning model with hierarchical layers));
presenting a result of recognition with respect to input data for verification (Fig. 1; [0058]-[0059]: "The recognition target visualization unit 14 displays recognition target information expressing the selected recognition target by superimposing on the ontology information. The recognition target visualization unit 14 calculates the recognition accuracy of the recognizer with respect to each recognition target selected by the recognition target selection unit 12 from evaluation data and generates the recognition target visualized information. … In other words, the recognition target visualization unit 14 calculates accuracy with respect to each recognition target. … The recognition target visualization unit 14 generates the recognition target visualized information visually expressing the recognition target of the recognizer based on the assembly of the input recognition target IDs and the ontology information. … The recognition target visualization unit 14 may superimpose the calculated recognition accuracy of each recognition target on the recognition target visualized information together with the concept information having the recognition target ID matching with the input recognition target ID as the recognition target information. The recognition target visualization unit 14 outputs the generated recognition target visualized information to the terminal apparatus 100" teaches a recognition target visualization unit 14 (presentation unit) that displays (presents) recognition target visualized information representing the results from the recognition unit (e.g. recognition accuracy for recognition target) for the input data);
obtaining information indicating a specific domain for which a recognition result needs to be improved in the input data for verification (Fig. 8; [0075]-[0079]: "A recognition training system 1a according to the present exemplary embodiment is described using a case in which the pre-trained recognizer is subjected to fine-tuning as an example. … the recognition training apparatus 10a adaptively trains the recognizer based on the operation information expressing feedback from the user to the recognition target visualized information. … Returning to the description of FIG. 8, the recognition training apparatus 10a is an apparatus for performing fine-tuning on the recognizer with respect to the specific domain. … The recognition target update unit 15 updates the recognition target based on the operation information indicating an operation by a person to the recognition target visualized information displayed on the display unit DS of the terminal apparatus 100. Specifically, the recognition target update unit 15 detects that the domain ID, the operation information “execution of fine-tuning”, and the assembly of the recognition target IDs are input from the terminal apparatus 100" teaches a recognition target update unit 15 (obtaining unit) that obtains operation information expressing feedback (e.g. needing improvement) for the recognition target visualized information (recognition result) to perform fine-tuning on the recognizer (machine learning model) with respect to the specific domain).
Hachiya et al. does not appear to explicitly teach performing training so as to optimize the machine learning model using data indicating a ground truth of an output from the machine learning model with respect to input data for training extracted regarding the specific domain, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model, and wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Li et al. teaches performing training so as to optimize the machine learning model using data indicating a ground truth of an output from the machine learning model with respect to input data for training extracted regarding the specific domain (Fig. 2; Fig. 3; [0060]: "the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module (learning unit) to train the domain-agnostic model (machine learning model) based on the loss (error) between a feature probability distribution 204 for the new domain data and ground truth data corresponding to the new domain (specific domain) input data (data indicating a ground truth of an output from the machine learning model regarding a specific domain of the input data)),
wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model (Fig. 2; Fig. 3; [0057]-[0060]: "To generate the local context vectors 316, the domain-specific model 118 implements a local domain classifier 322, which is representative of a fully-convolutional neural network configured to output a domain prediction map 324 having a same size (e.g., width and height) as the input data 302. In some implementations the local domain classifier 322 is trained using local loss 326. The local loss 326 is representative of any suitable type of loss algorithm for aligning low-level features, such as a least-squares loss. The local context vector 316 is extracted from a middle layer of the local domain classifier 322 … To generate the global context vectors 318, the domain-specific model 118 implements a global domain classifier 328, which is representative of a fully-convolutional neural network configured to predict a domain useable to describe the input data 302 … The global loss 330 is representative of a loss function that influences the global domain classifier 328 to ignore easy-to-classify examples (e.g., instances of the input data 302 being of a same domain as training data used to generate the domain-specific model 118) and focus on difficult-to-classify examples (e.g., instances of the input data 302 being of a different domain as training data used to generate the domain-specific model 118). In this manner, the global loss 330 may be representative of a variety of different known loss functions, such as a cross-entropy loss function, a focal loss function, and the like … By presuming any classified bounding box 320 output by the domain-specific model 118 to be unreliable when the input data 302 is representative of new domain data 122, the domain adaptation system 104 is configured to leverage information generated by the domain-specific model 118 to determine the loss function 206 for use in generating the domain-agnostic model 106 … the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on a local loss 326 (first loss) between the output of a local feature network (intermediate layer) of the model and the ground truth data corresponding to the new domain (specific domain) input data (specific domain ground truth data) and a global loss 330 (second loss) between the output of the global domain classifier of the model (output of a final layer of the machine learning model) and the ground truth data).
Hachiya et al. and Li et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate performing training so as to optimize the machine learning model using data indicating a ground truth of an output from the machine learning model with respect to input data for training extracted regarding the specific domain, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model as taught by Li et al. to the disclosed invention of Hachiya et al.
One of ordinary skill in the art would have been motivated to make this modification to "enable generation of a trained model configured to handle diverse input data without requiring the size and scope of training data otherwise necessitated by conventional approaches, thereby reducing an amount of computational and network resources used in training a model" (Li et al. [0024]).
Hachiya et al. in view of Li et al. does not appear to explicitly teach wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Kiraly et al. teaches wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on the generated classifications in a slice (response map) and the ground truth data. [0028]-[0029]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice ... the prostate tumor detection and classification results can be output by overlaying the generated benign tumor response maps and malignant tumor response maps on structural MRI images, such as T2-weighted MRI images. For example, 2D response maps generated for benign and malignant tumors can be overlaid on corresponding slices of the T2-weighted MRI image of the patient and displayed on a display device. In a possible embodiment, the generated benign tumor response maps and the generated malignant tumor response maps can be overlaid on slices of the T2-weighted MRI image using different colors to represent detected benign and malignant tumors in order to provide an easily distinguishable indication of the classification of each detected tumor" teaches that the generated classifications in a slice are response maps that are corresponding to a specific domain/classification).
Hachiya et al., Li et al., and Kiraly et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).
Hachiya et al. in view of Li et al. and further in view of Kiraly et al. does not appear to explicitly teach a response map of the input data generated, by a model that responds to a specific color of the input data.
However, Lin et al. teaches a response map of the input data generated, by a model that responds to a specific color of the input data ([0065]: "given a trained CNN, the CNN can process an input image to identify feature maps describing image features in locations in the image. The feature maps are created by processing the image with a convolutional neural network (CNN) which is trained to extract the image features (e.g., data describing objects) based on pixel values (e.g., color, grayscale value) of pixels within locations of the image. The CNN produces feature maps from the extracted image features" teaches the feature maps (response maps) being generated by a CNN (by a model) that responds to specific colors of the input image (input data)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a response map of the input data generated, by a model that responds to a specific color of the input data as taught by Lin et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Kiraly et al.
One of ordinary skill in the art would have been motivated to make this modification "to produce feature maps that better represent the features of the image, thus improving accuracy" (Lin et al. [0026]).
Regarding Claim 20,
Hachiya et al. teaches a non-transitory computer readable storage medium on which is stored a computer program for making a computer execute an information processing method for an information processing apparatus (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches a recognition training (information processing) apparatus. [0127]: "Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus" teaches a non-transitory computer-readable storage medium storing a program for making a computer execute computer executable instructions for performing a method that may be implemented using the embodied apparatus) operable to
train a machine learning model that has a hierarchical structure configured by a plurality of hierarchical layers and that is used for recognizing a recognition target in input data (Fig. 1; [0010]: "a recognition training apparatus includes a generation unit configured to generate relevance between a specific domain and a candidate of a recognition target based on ontology information expressing a concept structure of the specific domain, a selection unit configured to select the recognition target from the candidate of the recognition target based on the relevance generated by the generation unit, and a training unit configured to train a recognizer using training data regarding the recognition target selected by the selection unit" teaches the apparatus training a recognizer to recognize a recognition target in inputted data. Fig. 1; [0051]-[0052]: "The recognition training unit 13 trains the recognizer using training data regarding the selected recognition target … The recognition training unit 13 trains the recognizer which has the read moving image datum as an input and the read recognition target ID as an output. When the moving image data is a still image, and the recognition target is a type of an object, for example, a region-based CNN (R-CNN) … can be applied to the recognizer … Processing by the recognition training unit 13 when the R-CNN is used as the recognizer is specifically described here" teaches that the recognizer that is trained is a CNN (hierarchical machine learning model with hierarchical layers)).
Hachiya et al. does not appear to explicitly teach obtaining input data and data indicating a ground truth of an output from the machine learning model regarding the input data: and training the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model, and wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Li et al. teaches obtaining input data and data indicating a ground truth of an output from the machine learning model regarding the input data (Fig. 2; [0028]: "the input module 108 is configured to receive a new domain dataset 120 for use in training the domain-specific model 118 to generate the domain-agnostic model 106. The new domain dataset 120 includes new domain data 122 and ground truth data 124, which corresponds to ideal or optimal outputs to be produced by the model when processing the new domain data 122 according to an auxiliary task. In this manner, the new domain dataset 120 is useable by the domain adaptation system 104 to compute a loss function for generating the domain-agnostic model 106 based on the domain-specific model 118" teaches an input module (obtaining unit) for receiving new domain data (input data) and ground truth data corresponding to ideal outputs (data indicating a ground truth of an output from the machine learning model) from the model when processing new domain data (input data)); and
training the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data (Fig. 2; Fig. 3; [0060]: "the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module (learning unit) to train the domain-agnostic model (machine learning model) based on the loss (error) between a feature probability distribution 204 for the new domain data generated from the feature networks (intermediate layers) in the domain-specific model (output in an intermediate layer of the machine learning model with respect to the input data) and ground truth data corresponding to the new domain (specific domain) input data (data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data)),
wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model (Fig. 2; Fig. 3; [0057]-[0060]: "To generate the local context vectors 316, the domain-specific model 118 implements a local domain classifier 322, which is representative of a fully-convolutional neural network configured to output a domain prediction map 324 having a same size (e.g., width and height) as the input data 302. In some implementations the local domain classifier 322 is trained using local loss 326. The local loss 326 is representative of any suitable type of loss algorithm for aligning low-level features, such as a least-squares loss. The local context vector 316 is extracted from a middle layer of the local domain classifier 322 … To generate the global context vectors 318, the domain-specific model 118 implements a global domain classifier 328, which is representative of a fully-convolutional neural network configured to predict a domain useable to describe the input data 302 … The global loss 330 is representative of a loss function that influences the global domain classifier 328 to ignore easy-to-classify examples (e.g., instances of the input data 302 being of a same domain as training data used to generate the domain-specific model 118) and focus on difficult-to-classify examples (e.g., instances of the input data 302 being of a different domain as training data used to generate the domain-specific model 118). In this manner, the global loss 330 may be representative of a variety of different known loss functions, such as a cross-entropy loss function, a focal loss function, and the like … By presuming any classified bounding box 320 output by the domain-specific model 118 to be unreliable when the input data 302 is representative of new domain data 122, the domain adaptation system 104 is configured to leverage information generated by the domain-specific model 118 to determine the loss function 206 for use in generating the domain-agnostic model 106 … the feature representation module 110 extracts the feature representation for input data 302 using a different component of the domain-specific model 118, such as the global feature network 308. The feature representation 202 is then useable by the domain transfer module 112 to generate a feature probability distribution 204 for the input data 302, illustrated in the example of FIG. 3 as graphically depicting probability distributions for each pixel of the input data 302 as corresponding to a different feature channel included in the feature representation 202 (e.g., vector graphics, raster graphics, or text). The feature probability distribution 204 is then comparable by the loss module 114 to ground truth data corresponding to the input data 302 to compute loss function 206, which is useable by the training module 116 to fine-tune weights of the domain-agnostic model 106. In an example implementation where the domain-agnostic model 106 is configured using the architecture of the domain-specific model 118 as illustrated in FIG. 3, fine-tuning weights of the domain-agnostic model 106 may include fine-tuning weights associated with one or more components, such as the local feature network 304, the global feature network 308, the detection network 312, the local domain classifier 322, or the global domain classifier 328" teaches a training module to train the domain-agnostic model (machine learning model) based on a local loss 326 (first loss) between the output of a local feature network (intermediate layer) of the model and the ground truth data corresponding to the new domain (specific domain) input data (specific domain ground truth data) and a global loss 330 (second loss) between the output of the global domain classifier of the model (output of a final layer of the machine learning model) and the ground truth data).
Hachiya et al. and Li et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate obtaining input data and data indicating a ground truth of an output from the machine learning model regarding the input data: and training the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model as taught by Li et al. to the disclosed invention of Hachiya et al.
One of ordinary skill in the art would have been motivated to make this modification to "enable generation of a trained model configured to handle diverse input data without requiring the size and scope of training data otherwise necessitated by conventional approaches, thereby reducing an amount of computational and network resources used in training a model" (Li et al. [0024]).
Hachiya et al. in view of Li et al. does not appear to explicitly teach wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
However, Kiraly et al. teaches wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on the generated classifications in a slice (response map) and the ground truth data. [0028]-[0029]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice ... the prostate tumor detection and classification results can be output by overlaying the generated benign tumor response maps and malignant tumor response maps on structural MRI images, such as T2-weighted MRI images. For example, 2D response maps generated for benign and malignant tumors can be overlaid on corresponding slices of the T2-weighted MRI image of the patient and displayed on a display device. In a possible embodiment, the generated benign tumor response maps and the generated malignant tumor response maps can be overlaid on slices of the T2-weighted MRI image using different colors to represent detected benign and malignant tumors in order to provide an easily distinguishable indication of the classification of each detected tumor" teaches that the generated classifications in a slice are response maps that are corresponding to a specific domain/classification).
Hachiya et al., Li et al., and Kiraly et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).
Hachiya et al. in view of Li et al. and further in view of Kiraly et al. does not appear to explicitly teach a response map of the input data generated, by a model that responds to a specific color of the input data.
However, Lin et al. teaches a response map of the input data generated, by a model that responds to a specific color of the input data ([0065]: "given a trained CNN, the CNN can process an input image to identify feature maps describing image features in locations in the image. The feature maps are created by processing the image with a convolutional neural network (CNN) which is trained to extract the image features (e.g., data describing objects) based on pixel values (e.g., color, grayscale value) of pixels within locations of the image. The CNN produces feature maps from the extracted image features" teaches the feature maps (response maps) being generated by a CNN (by a model) that responds to specific colors of the input image (input data)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a response map of the input data generated, by a model that responds to a specific color of the input data as taught by Lin et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Kiraly et al.
One of ordinary skill in the art would have been motivated to make this modification "to produce feature maps that better represent the features of the image, thus improving accuracy" (Lin et al. [0026]).
Regarding Claim 21,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 13.
In addition, Kiraly et al. further teaches wherein the specific domain ground truth data is obtained by multiplying the response map and the ground truth data (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on generating a Gaussian distribution (e.g. includes multiplying) using the generated classifications in a slice (response map) and the ground truth data. [0028]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice" teaches that the generated classifications in a slice are response maps corresponding to a specific domain/classification).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is obtained by multiplying the response map and the ground truth data as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).Regarding Claim 22,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 13.
In addition, Kiraly et al. further teaches wherein the specific domain ground truth data is calculated by integrating a first response map of the input data by a first model that responds to a first domain of the input data and a second response map of the input data by a second model that responds to a second domain of the input data using weights (Fig .1; [0027]: "Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location" teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on generating a Gaussian distribution (e.g. includes multiplying) using the generated classifications in a slice (response maps) and the ground truth data. [0028]: "the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice" teaches that the generated classifications in a slice are response maps corresponding to a specific domain/classification. [0024]: "the trained image-to-image convolutional encoder-decoder has two output channels corresponding to benign and malignant prostate tumors, respectively. In this case, the trained image-to-image convolutional encoder-decoder inputs the mpMRI images and outputs a first response map showing detected locations of prostate tumors classified as benign and a second response map showing detected locations or prostate tumors classified as malignant. In particular, the first response map shows the detected locations of tumors classified as benign with a Gaussian intensity distribution centered at each detected location and the second response map shows the detected locations of tumors classified as malignant with a Gaussian intensity distribution centered at each detected location" teaches first and second response maps of the input data that respectively, by a model, respond to a first domain (e.g. benign tumors) and a second domain (e.g. malignant tumors)).
Hachiya et al., Li et al., Kiraly et al., and Lin et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the specific domain ground truth data is calculated by integrating a first response map of the input data by a first model that responds to a first domain of the input data and a second response map of the input data by a second model that responds to a second domain of the input data using weights as taught by Kiraly et al. to the disclosed invention of Hachiya et al. in view of Li et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "simultaneously perform detection and classification of … multiple classes ... within a series of multi-parametric input images" (Kiraly et al. [0018]).
Claims 2 and 7-10 are rejected under 35 U.S.C. 103 as being unpatentable over Hachiya et al. (US 2017/0206437 A1) in view of Li et al. (US 2021/0334664 A1) in view of Kiraly et al. (US 2018/0240233 A1) in view of Lin et al. (US 2018/0260698 A1) and further in view of Castillo et al. (US 2021/0243362 A1).
Regarding Claim 2,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
Hachiya et al. in view of Li et al. and further in view of Kiraly et al. does not appear to explicitly teach wherein the instructions further cause the information processing apparatus to extract a specific domain region from the input data.
However, Castillo et al. teaches wherein the instructions further cause the information processing apparatus to extract a specific domain region from the input data (Fig. 15; [0120]: "While bounding boxes are a simple and straightforward tool for analyzing an image position within a display, segmentation masks may provide more direct actionable feedback. FIG. 15 illustrates a segmentation mask 1502 overlaid on subject 602. Segmentation mask 1502 may be generated by a classifier or object identification module of an image capture device; MobileNet is an example of a classifier that runs on small devices. The classifier may be trained separately to identify specific objects within an image and provide a mask to that object. The contours of a segmentation mask are typically irregular at the pixel determination for where an object begins and the rest of the scene ends and can be noisy. As such, mask 1502 need not be, and rarely is, a perfect overlay of subject 602" teaches that an object identification module can extract a segmentation mask (specific domain region) for a specific object (specific domain) from an image (input data)).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Castillo et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the instructions further cause the information processing apparatus to extract a specific domain region from the input data as taught by Castillo et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "improve the quality of depth information that can be extracted from the captured image" (Castillo et al. [0065]).
Regarding Claim 7,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 6.
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. does not appear to explicitly teach wherein information indicating a region belonging to the specific domain in the input data for verification is obtained.
However, Castillo et al. teaches wherein information indicating a region belonging to the specific domain in the input data for verification is obtained (Fig. 15; [0120]: "While bounding boxes are a simple and straightforward tool for analyzing an image position within a display, segmentation masks may provide more direct actionable feedback. FIG. 15 illustrates a segmentation mask 1502 overlaid on subject 602. Segmentation mask 1502 may be generated by a classifier or object identification module of an image capture device; MobileNet is an example of a classifier that runs on small devices. The classifier may be trained separately to identify specific objects within an image and provide a mask to that object. The contours of a segmentation mask are typically irregular at the pixel determination for where an object begins and the rest of the scene ends and can be noisy. As such, mask 1502 need not be, and rarely is, a perfect overlay of subject 602" teaches that an object identification module can extract a segmentation mask (specific domain region) for a specific object (specific domain) from an image (input data) for verification).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Castillo et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein information indicating a region belonging to the specific domain in the input data for verification is obtained as taught by Castillo et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "improve the quality of depth information that can be extracted from the captured image" (Castillo et al. [0065]).
Regarding Claim 8,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. does not appear to explicitly teach wherein the instructions further cause the information processing apparatus to create a model that extracts a specific domain region from a feature amount in a region belonging to the specific domain in the input data.
However, Castillo et al. teaches wherein the instructions further cause the information processing apparatus to create a model that extracts a specific domain region from a feature amount in a region belonging to the specific domain in the input data (Fig. 15; [0120]: "While bounding boxes are a simple and straightforward tool for analyzing an image position within a display, segmentation masks may provide more direct actionable feedback. FIG. 15 illustrates a segmentation mask 1502 overlaid on subject 602. Segmentation mask 1502 may be generated by a classifier or object identification module of an image capture device; MobileNet is an example of a classifier that runs on small devices. The classifier may be trained separately to identify specific objects within an image and provide a mask to that object. The contours of a segmentation mask are typically irregular at the pixel determination for where an object begins and the rest of the scene ends and can be noisy. As such, mask 1502 need not be, and rarely is, a perfect overlay of subject 602" teaches that an object identification module can create a trained classifier to extract a segmentation mask (specific domain region) for a specific object (specific domain) from an image (input data) for verification. [0079]: "Detecting the target physical structure can include performing one or more image segmentation techniques, which include inputting a 2D image into a trained classifier to detect pixels relating to the target physical structure, such as a house. When the target physical structure is detected, the intra-image parameter evaluation system 250 can determine the dimensions of a bounding box and render the bounding box around the target physical structure" teaches that the classifier (model) determines the bounding box for the mask (region) based on pixels relating (i.e. pixels relating to features) to the target (specific domain)).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Castillo et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the instructions further cause the information processing apparatus to create a model that extracts a specific domain region from a feature amount in a region belonging to the specific domain in the input data as taught by Castillo et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "improve the quality of depth information that can be extracted from the captured image" (Castillo et al. [0065]).
Regarding Claim 9,
Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al. teaches information processing apparatus according to claim 7.
In addition, Castillo et al. further teaches wherein the region belonging to the specific domain is at least one of a region in which a recognition target is present but is erroneously not recognized and a region in which a recognition target is not present but is erroneously recognized (Fig. 15; [0120]-[0121]: "While bounding boxes are a simple and straightforward tool for analyzing an image position within a display, segmentation masks may provide more direct actionable feedback. FIG. 15 illustrates a segmentation mask 1502 overlaid on subject 602. Segmentation mask 1502 may be generated by a classifier or object identification module of an image capture device; MobileNet is an example of a classifier that runs on small devices. The classifier may be trained separately to identify specific objects within an image and provide a mask to that object. The contours of a segmentation mask are typically irregular at the pixel determination for where an object begins and the rest of the scene ends and can be noisy. As such, mask 1502 need not be, and rarely is, a perfect overlay of subject 602 … This noisy overlay still provides a better approximation of the subject's true presence in the display. While a bounding box ensures all pixels of a subject are within, there are still many pixels within a bounding box geometry that do not depict the subject" teaches that an object identification module can extract a segmentation mask (specific domain region) for a specific object (specific domain) from an image (input data) for verification, the mask (region) recognizing some pixels as part of the object (specific domain) when they do not depict the object (recognition target is not present but is erroneously recognized)).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Castillo et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the region belonging to the specific domain is at least one of a region in which a recognition target is present but is erroneously not recognized and a region in which a recognition target is not present but is erroneously recognized as taught by Castillo et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "improve the quality of depth information that can be extracted from the captured image" (Castillo et al. [0065]).
Regarding Claim 10,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. does not appear to explicitly teach wherein the instructions further cause the information processing apparatus to: evaluate a degree of a contribution to a final output of the machine learning model for each channel of the intermediate layer; and select a channel to be used for training of the machine learning model from a plurality of channels of the intermediate layer based on the contribution.
However, Castillo et al. teaches wherein the instructions further cause the information processing apparatus to: evaluate a degree of a contribution to a final output of the machine learning model for each channel of the intermediate layer (Fig. 1; [0183]-[0184]: "a channel can be an activation map for data in an image frame (pre- or post-capture) indicating a model's prediction that a pixel in the image frame is attributable to a particular classification of a broader segmentation mask. The activation maps can be, then, an inverse representation, or single slice, of a segmentation mask trained for multiple classifications. By selectively isolating or combining single activation maps, new semantic information, masks, and bounding boxes can be created for sub-structures or subfeatures in the scene within the image frame and guidance prompts provided to optimize framing for those elements (e.g., the sub-structures or the subfeatures) … a neural network model comprises a plurality of layers for classifying pixels as subfeatures within an image. A final convolution layer separates out, into desired channels or subchannels, outputs representing only a single classification of the model's constituent elements. This enables feature representations across the image to influence prediction of subfeatures, while still maintaining a layer optimized for a specific feature. In other words, a joint prediction of multiple classes is enabled by this system (e.g., by server 120 and its components) … each subchannel in the final layer output is compared during training to a ground truth image of those same classified features and any error in each subchannel is propagated back through the network. This results in a trained model that outputs N channels of segmentation masks corresponding to target subfeatures of the aggregate mask. Merely for illustrative purposes, the six masks depicted among group 3802 reflect a six feature output of such a trained model. Each activation map in these channels is a component of an overall segmentation mask (or as aggregated a segmentation map of constituent segmentation masks)" teaches a server 120 and its components that evaluates that influence of prediction (degree of contribution) of sub features to a joint prediction of multiple classes for a final classification output of the machine learning model for each channel of the final convolution layer (intermediate layer)); and
select a channel to be used for training of the machine learning model from a plurality of channels of the intermediate layer based on the contribution (Fig. 1; [0183]-[0184]: "a channel can be an activation map for data in an image frame (pre- or post-capture) indicating a model's prediction that a pixel in the image frame is attributable to a particular classification of a broader segmentation mask. The activation maps can be, then, an inverse representation, or single slice, of a segmentation mask trained for multiple classifications. By selectively isolating or combining single activation maps, new semantic information, masks, and bounding boxes can be created for sub-structures or subfeatures in the scene within the image frame and guidance prompts provided to optimize framing for those elements (e.g., the sub-structures or the subfeatures) … a neural network model comprises a plurality of layers for classifying pixels as subfeatures within an image. A final convolution layer separates out, into desired channels or subchannels, outputs representing only a single classification of the model's constituent elements. This enables feature representations across the image to influence prediction of subfeatures, while still maintaining a layer optimized for a specific feature. In other words, a joint prediction of multiple classes is enabled by this system (e.g., by server 120 and its components) … each subchannel in the final layer output is compared during training to a ground truth image of those same classified features and any error in each subchannel is propagated back through the network. This results in a trained model that outputs N channels of segmentation masks corresponding to target subfeatures of the aggregate mask. Merely for illustrative purposes, the six masks depicted among group 3802 reflect a six feature output of such a trained model. Each activation map in these channels is a component of an overall segmentation mask (or as aggregated a segmentation map of constituent segmentation masks)" teaches a server 120 and its components that a channel that is an activation map for data in an image indicating a model's prediction (e.g. based on contribution) that a pixel in the image frame is attributable to a particular classification may be selected and used for training the model).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Castillo et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the instructions further cause the information processing apparatus to: evaluate a degree of a contribution to a final output of the machine learning model for each channel of the intermediate layer; and select a channel to be used for training of the machine learning model from a plurality of channels of the intermediate layer based on the contribution as taught by Castillo et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "improve the quality of depth information that can be extracted from the captured image" (Castillo et al. [0065]).
Claims 3-5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Hachiya et al. (US 2017/0206437 A1) in view of Li et al. (US 2021/0334664 A1) in view of Kiraly et al. (US 2018/0240233 A1) in view of Lin et al. (US 2018/0260698 A1) in view of Castillo et al. (US 2021/0243362 A1) and further in view of Kuroda et al. (US 2018/0307946 A1).
Regarding Claim 3,
Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al. teaches information processing apparatus according to claim 2.
Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al. does not appear to explicitly teach wherein the instructions further cause the information processing apparatus to create the data indicating the ground truth regarding the specific domain of input data from the data indicating the ground truth of the output from the machine learning model regarding the input data in the specific domain region.
However, Kuroda et al. teaches wherein the instructions further cause the information processing apparatus to create the data indicating the ground truth regarding the specific domain of input data from the data indicating the ground truth of the output from the machine learning model regarding the input data in the specific domain region (Fig. 4; Fig 6; [0095]-[0096]: "The training data creation unit 34 associates the input data acquired by the input data acquisition unit 31 with the evaluation for each label acquired by the evaluation acquisition unit 32 to create the training data. The training data creation unit 34 may gather the input data and the evaluation for each label into one piece data as the training data, or use a table to associate the input data with the evaluation for each label … FIG. 6 illustrates an example of the training data. As shown in (A) of FIG. 6, each piece of input data T1 to TN (N is a natural number) is associated with a plurality of labels. Here, each piece of the input data T1 to TN is associated with three labels. For example, a first label L1 is a label representing that the content of the image is a “dog”, a second label L2 is a label representing that the content of the image is a “person”, and a third label L3 is a label representing that the content of the image is a “flower”. The training data creation unit 34 makes associations of the evaluations of all labels for each input data. For example, assume that the input data T1 is an image of a dog, in which no person appears. In this case, the evaluation of the first label L1 of positive is stored in the table, and the evaluation of the second label L2 of negative is stored in the table. Note that in a case that whether or not a flower appears in the input data T3 is unknown, or in a case that whether or not a flower appears is not determined (a case that the evaluation acquisition unit 32 cannot acquire the evaluation), the ignorable evaluation is determined, and the evaluation of the third label L3 of ignoring is stored in the table. In this way, each piece of the input data T1 to TN is associated with the evaluations of the respective labels. Note that the evaluation may be indicated by the score, for example, the positive evaluation is “1”, and the negative evaluation is “0”. The score indicating the evaluation of the input data is referred to as a ground-truth score" teaches a training data creation unit 34 (first creation unit) to create training data for input data for each specific label (data indicating the ground truth regarding the specific domain of input data) from the evaluation data for each label, the evaluation being a ground truth score (data indicating the ground truth of the output from the machine learning model)).
Hachiya et al., Li et al., Kiraly et al., Lin et al., Castillo et al., and Kuroda et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the instructions further cause the information processing apparatus to create the data indicating the ground truth regarding the specific domain of input data from the data indicating the ground truth of the output from the machine learning model regarding the input data in the specific domain region. as taught by Kuroda et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al.
One of ordinary skill in the art would have been motivated to make this modification to "prevent training based on the incorrect evaluation from being performed but also perform training based on a correct evaluation, and as a result, the accuracy performance of the recognition unit can be improved" (Kuroda et al. [0016]).
Regarding Claim 4,
Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. in view of Castillo et al. and further in view of Kuroda et al. teaches information processing apparatus according to claim 3.
In addition, Kuroda et al. further teaches wherein the data indicating the ground truth regarding the specific domain of the input data is created from ground truth data indicating whether or not each element of the input data in the specific domain region belongs to a specific class (Fig. 4; Fig 6; [0095]-[0096]: "The training data creation unit 34 associates the input data acquired by the input data acquisition unit 31 with the evaluation for each label acquired by the evaluation acquisition unit 32 to create the training data. The training data creation unit 34 may gather the input data and the evaluation for each label into one piece data as the training data, or use a table to associate the input data with the evaluation for each label … FIG. 6 illustrates an example of the training data. As shown in (A) of FIG. 6, each piece of input data T1 to TN (N is a natural number) is associated with a plurality of labels. Here, each piece of the input data T1 to TN is associated with three labels. For example, a first label L1 is a label representing that the content of the image is a “dog”, a second label L2 is a label representing that the content of the image is a “person”, and a third label L3 is a label representing that the content of the image is a “flower”. The training data creation unit 34 makes associations of the evaluations of all labels for each input data. For example, assume that the input data T1 is an image of a dog, in which no person appears. In this case, the evaluation of the first label L1 of positive is stored in the table, and the evaluation of the second label L2 of negative is stored in the table. Note that in a case that whether or not a flower appears in the input data T3 is unknown, or in a case that whether or not a flower appears is not determined (a case that the evaluation acquisition unit 32 cannot acquire the evaluation), the ignorable evaluation is determined, and the evaluation of the third label L3 of ignoring is stored in the table. In this way, each piece of the input data T1 to TN is associated with the evaluations of the respective labels. Note that the evaluation may be indicated by the score, for example, the positive evaluation is “1”, and the negative evaluation is “0”. The score indicating the evaluation of the input data is referred to as a ground-truth score" teaches a training data creation unit 34 (first creation unit) to create training data for input data for each specific label (data indicating the ground truth regarding the specific domain of input data) from the evaluation data for each label, the evaluation being a ground truth score indicating whether or not each element of the input data belongs to a specific class).
Hachiya et al., Li et al., Kiraly et al., Lin et al., Castillo et al., and Kuroda et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the data indicating the ground truth regarding the specific domain of the input data is created from ground truth data indicating whether or not each element of the input data in the specific domain region belongs to a specific class as taught by Kuroda et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al.
One of ordinary skill in the art would have been motivated to make this modification to "prevent training based on the incorrect evaluation from being performed but also perform training based on a correct evaluation, and as a result, the accuracy performance of the recognition unit can be improved" (Kuroda et al. [0016]).
Regarding Claim 5,
Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al. teaches information processing apparatus according to claim 2.
In addition, Castillo et al. further teaches wherein the instructions further cause the information processing apparatus to extract first and second domain regions from the input data (Fig. 15; [0120]: "While bounding boxes are a simple and straightforward tool for analyzing an image position within a display, segmentation masks may provide more direct actionable feedback. FIG. 15 illustrates a segmentation mask 1502 overlaid on subject 602. Segmentation mask 1502 may be generated by a classifier or object identification module of an image capture device; MobileNet is an example of a classifier that runs on small devices. The classifier may be trained separately to identify specific objects within an image and provide a mask to that object. The contours of a segmentation mask are typically irregular at the pixel determination for where an object begins and the rest of the scene ends and can be noisy. As such, mask 1502 need not be, and rarely is, a perfect overlay of subject 602" teaches that an object identification module (extraction unit) can extract a segmentation mask (specific domain region) for a specific object (specific domain) from an image (input data). Fig. 23A; [0148]: "FIG. 23A illustrates this scenario, with two masks 2302 and 2304 both present for the single house in the frame, divided by occluding object tree 2312. These masks 2302 and 2304 may be referred to as “small neighbor” masks. In some embodiments, at block 2401, the largest segmentation mask among a plurality of small neighbors in the frame is selected. In some embodiments, the segmentation mask with a centroid closest to the center of the display is selected. Referring to FIG. 23A, neighbor mask 2302 is likely to be selected as the initial segmentation mask as its pixel area is larger compared to neighbor mask 2304, and its centroid is closer to the center of display 2320" teaches extracting multiple segmentation masks (first and second domain regions) for different objects (e.g. house and tree) from an image (input data)).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Castillo et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the instructions further cause the information processing apparatus to extract first and second domain regions from the input data as taught by Castillo et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "improve the quality of depth information that can be extracted from the captured image" (Castillo et al. [0065]).
Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al. does not appear to explicitly teach wherein the data indicating the ground truth regarding the specific domain of the input data is a combination of data indicating a ground truth of an output from the machine learning model regarding the input data in the first domain region and data indicating a ground truth of an output from the machine learning model regarding the input data in the second domain region.
However, Kuroda et al. teaches wherein the data indicating the ground truth regarding the specific domain of the input data is a combination of data indicating a ground truth of an output from the machine learning model regarding the input data in the first domain region and data indicating a ground truth of an output from the machine learning model regarding the input data in the second domain region (Fig. 4; Fig 6; [0095]-[0096]: "The training data creation unit 34 associates the input data acquired by the input data acquisition unit 31 with the evaluation for each label acquired by the evaluation acquisition unit 32 to create the training data. The training data creation unit 34 may gather the input data and the evaluation for each label into one piece data as the training data, or use a table to associate the input data with the evaluation for each label … FIG. 6 illustrates an example of the training data. As shown in (A) of FIG. 6, each piece of input data T1 to TN (N is a natural number) is associated with a plurality of labels. Here, each piece of the input data T1 to TN is associated with three labels. For example, a first label L1 is a label representing that the content of the image is a “dog”, a second label L2 is a label representing that the content of the image is a “person”, and a third label L3 is a label representing that the content of the image is a “flower”. The training data creation unit 34 makes associations of the evaluations of all labels for each input data. For example, assume that the input data T1 is an image of a dog, in which no person appears. In this case, the evaluation of the first label L1 of positive is stored in the table, and the evaluation of the second label L2 of negative is stored in the table. Note that in a case that whether or not a flower appears in the input data T3 is unknown, or in a case that whether or not a flower appears is not determined (a case that the evaluation acquisition unit 32 cannot acquire the evaluation), the ignorable evaluation is determined, and the evaluation of the third label L3 of ignoring is stored in the table. In this way, each piece of the input data T1 to TN is associated with the evaluations of the respective labels. Note that the evaluation may be indicated by the score, for example, the positive evaluation is “1”, and the negative evaluation is “0”. The score indicating the evaluation of the input data is referred to as a ground-truth score" teaches a training data creation unit 34 (first creation unit) to create training data for input data for each specific label (data indicating the ground truth regarding the specific domain of input data) from the evaluation data for each label, the evaluation being a ground truth score indicating a ground truth output for each label (domain) (e.g. ground truth for first domain and ground truth for second domain)).
Hachiya et al., Li et al., Kiraly et al., Lin et al., Castillo et al., and Kuroda et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the data indicating the ground truth regarding the specific domain of the input data is a combination of data indicating a ground truth of an output from the machine learning model regarding the input data in the first domain region and data indicating a ground truth of an output from the machine learning model regarding the input data in the second domain region as taught by Kuroda et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al.
One of ordinary skill in the art would have been motivated to make this modification to "prevent training based on the incorrect evaluation from being performed but also perform training based on a correct evaluation, and as a result, the accuracy performance of the recognition unit can be improved" (Kuroda et al. [0016]).
Regarding Claim 12,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches information processing apparatus according to claim 1.
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. does not appear to explicitly teach wherein a combination of: at least one intermediate layer of the machine learning model, and at least one of the specific domain and a specific class is decided, and the specific class is referenced to create the data indicating the ground truth regarding the specific domain of the input data from ground truth data indicating whether or not each element of the input data belongs to the specific class.
However, Castillo et al. teaches wherein a combination of: at least one intermediate layer of the machine learning model, and at least one of the specific domain and a specific class is decided (Fig. 1; [0183]-[0184]: "a channel can be an activation map for data in an image frame (pre- or post-capture) indicating a model's prediction that a pixel in the image frame is attributable to a particular classification of a broader segmentation mask. The activation maps can be, then, an inverse representation, or single slice, of a segmentation mask trained for multiple classifications. By selectively isolating or combining single activation maps, new semantic information, masks, and bounding boxes can be created for sub-structures or subfeatures in the scene within the image frame and guidance prompts provided to optimize framing for those elements (e.g., the sub-structures or the subfeatures) … a neural network model comprises a plurality of layers for classifying pixels as subfeatures within an image. A final convolution layer separates out, into desired channels or subchannels, outputs representing only a single classification of the model's constituent elements. This enables feature representations across the image to influence prediction of subfeatures, while still maintaining a layer optimized for a specific feature. In other words, a joint prediction of multiple classes is enabled by this system (e.g., by server 120 and its components) … each subchannel in the final layer output is compared during training to a ground truth image of those same classified features and any error in each subchannel is propagated back through the network. This results in a trained model that outputs N channels of segmentation masks corresponding to target subfeatures of the aggregate mask. Merely for illustrative purposes, the six masks depicted among group 3802 reflect a six feature output of such a trained model. Each activation map in these channels is a component of an overall segmentation mask (or as aggregated a segmentation map of constituent segmentation masks)" teaches a server 120 and its components (learning unit) that evaluates each channel of the final convolution layer (intermediate layer) and its classification prediction (e.g. for a specific class) with a ground truth to determine an error for back propagation during training).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Castillo et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein a combination of: at least one intermediate layer of the machine learning model, and at least one of the specific domain and a specific class is decided as taught by Castillo et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification to "improve the quality of depth information that can be extracted from the captured image" (Castillo et al. [0065]).
Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al. does not appear to explicitly teach the specific class is referenced to create the data indicating the ground truth regarding the specific domain of the input data from ground truth data indicating whether or not each element of the input data belongs to the specific class.
However, Kuroda et al. teaches the specific class is referenced to create the data indicating the ground truth regarding the specific domain of the input data from ground truth data indicating whether or not each element of the input data belongs to the specific class (Fig. 4; Fig 6; [0095]-[0096]: "The training data creation unit 34 associates the input data acquired by the input data acquisition unit 31 with the evaluation for each label acquired by the evaluation acquisition unit 32 to create the training data. The training data creation unit 34 may gather the input data and the evaluation for each label into one piece data as the training data, or use a table to associate the input data with the evaluation for each label … FIG. 6 illustrates an example of the training data. As shown in (A) of FIG. 6, each piece of input data T1 to TN (N is a natural number) is associated with a plurality of labels. Here, each piece of the input data T1 to TN is associated with three labels. For example, a first label L1 is a label representing that the content of the image is a “dog”, a second label L2 is a label representing that the content of the image is a “person”, and a third label L3 is a label representing that the content of the image is a “flower”. The training data creation unit 34 makes associations of the evaluations of all labels for each input data. For example, assume that the input data T1 is an image of a dog, in which no person appears. In this case, the evaluation of the first label L1 of positive is stored in the table, and the evaluation of the second label L2 of negative is stored in the table. Note that in a case that whether or not a flower appears in the input data T3 is unknown, or in a case that whether or not a flower appears is not determined (a case that the evaluation acquisition unit 32 cannot acquire the evaluation), the ignorable evaluation is determined, and the evaluation of the third label L3 of ignoring is stored in the table. In this way, each piece of the input data T1 to TN is associated with the evaluations of the respective labels. Note that the evaluation may be indicated by the score, for example, the positive evaluation is “1”, and the negative evaluation is “0”. The score indicating the evaluation of the input data is referred to as a ground-truth score" teaches creating training data for input data for each specific label (data indicating the ground truth regarding the specific domain of input data) from the evaluation data for each label, the evaluation being a ground truth score indicating whether or not each element of the input data belongs to a specific class).
Hachiya et al., Li et al., Kiraly et al., Lin et al., Castillo et al., and Kuroda et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the specific class is referenced to create the data indicating the ground truth regarding the specific domain of the input data from ground truth data indicating whether or not each element of the input data belongs to the specific class as taught by Kuroda et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. in view of Lin et al. and further in view of Castillo et al.
One of ordinary skill in the art would have been motivated to make this modification to "prevent training based on the incorrect evaluation from being performed but also perform training based on a correct evaluation, and as a result, the accuracy performance of the recognition unit can be improved" (Kuroda et al. [0016]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Hachiya et al. (US 2017/0206437 A1) in view of Li et al. (US 2021/0334664 A1) in view of Kiraly et al. (US 2018/0240233 A1) in view of Lin et al. (US 2018/0260698 A1) and further in view of Zoph et al. ("Neural Architecture Search with Reinforcement Learning").
Regarding Claim 11,
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. teaches the information processing apparatus according to claim 1.
Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al. does not appear to explicitly teach wherein the machine learning model is trained by reinforcement learning so as to maximize an accuracy for at least one of a recognition accuracy for input data for verification and a recognition accuracy for a specific domain in the input data for verification.
However, Zoph et al. teaches wherein the machine learning model is trained by reinforcement learning so as to maximize an accuracy for at least one of a recognition accuracy for input data for verification and a recognition accuracy for a specific domain in the input data for verification (Abstract: "we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy" teaches neural network model training using reinforcement learning to maximize accuracy. Fig. 1; Section 1, second - third paragraphs: "This paper presents Neural Architecture Search, a gradient-based method for finding good architectures (see Figure 1) . Our work is based on the observation that the structure and connectivity of a neural network can be typically specified by a variable-length string. It is therefore possible to use a recurrent network – the controller – to generate such string. Training the network specified by the string – the “child network” – on the real data will result in an accuracy on a validation set. Using this accuracy as the reward signal, we can compute the policy gradient to update the controller. As a result, in the next iteration, the controller will give higher probabilities to architectures that receive high accuracies. In other words, the controller will learn to improve its search over time. Our experiments show that Neural Architecture Search can design good models from scratch, an achievement considered not possible with other methods. On image recognition with CIFAR-10, Neural Architecture Search can find a novel ConvNet model that is better than most human-invented architectures. Our CIFAR-10 model achieves a 3.65 test set error, while being 1.05x faster than the current best model" teaches that the machine learning model is updated (trained) by using a reward (e.g. via reinforcement learning) based on maximizing accuracy for a recognition accuracy for real data (input data) for validation (verification)).
Hachiya et al., Li et al., Kiraly et al., Lin et al., and Zoph et al. are analogous to the claimed invention because they are directed machine learning image classification.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate wherein the machine learning model is trained by reinforcement learning so as to maximize an accuracy for at least one of a recognition accuracy for input data for verification and a recognition accuracy for a specific domain in the input data for verification as taught by Zoph et al. to the disclosed invention of Hachiya et al. in view of Li et al. in view of Kiraly et al. and further in view of Lin et al.
One of ordinary skill in the art would have been motivated to make this modification "to maximize the expected accuracy of the generated architectures on a validation set" (Zoph et al. Abstract).
Response to Arguments
Applicant’s arguments, filed 07/31/2026, with respect to the claim objections have been fully considered and are persuasive. Therefore, the objections rejections have been withdrawn. However, upon further consideration, a new ground(s) of objection is made in view of the claim amendments filed 07/31/2026.
Applicant’s arguments, filed 07/31/2026, with respect to the claim rejections under 35 U.S.C. 112(b) have been fully considered and are persuasive. Therefore, the 35 U.S.C. 112(b) rejections have been withdrawn.
Applicant's arguments, filed 07/31/2026, with respect to the 35 U.S.C. 103 prior art rejections to claims 1-22 have been fully considered but they are not persuasive. Applicant asserts “Claims 1, 6 and 13 to 22 were rejected under 35 U.S.C. §103 as being obvious over U.S. Publication No. 2017/0206437 (Hachiya) in view of U.S. Publication No. 2021/0334664 (Li) and U.S. Publication No. 2018/0240233 (Kiraly). Claims 2 and 7 to 10 were rejected under §@103 over Hachiya in view of Li and Kiraly and further in view of U.S. Publication No. 202 1/0243362 (Castillo). Claims 3 to 5 and 12 were rejected under §@103 over Hachiya in view of Li, Kiraly, and Castillo, and further in view of U.S. Publication No. 2018/0307946 (Kuroda). Claim 11 was rejected under §@103 over Hachiya in view of Li and Kiraly and further in view of NPL "Neural Architecture Search with Reinforcement Learning to(Zoph). The rejections are respectfully traversed and the Applicant requests that the Examiner reconsider and withdraw the rejections in light of the following comments.
Referring to the claim language, independent Claim 1 is directed to an information processing apparatus operable to train a machine learning model that has a hierarchical structure configured by a plurality of hierarchical layers and that is used for recognizing a recognition target in input data, the apparatus comprising: at least one processor; and at least one memory having stored thereon instructions which, when executed by the at least one processor, cause the information processing apparatus at least to: obtain input data and data indicating a ground truth of an output from the machine learning model regarding the input data; and train the machine learning model based on an error between the data indicating the ground truth of the output from the machine learning model regarding a specific domain of the input data and at least one output in an intermediate layer of the machine learning model with respect to the input data, wherein the machine learning model is trained based on (1) a first loss between (i) a specific domain ground truth data corresponding to the input data and (ii) at least one output at an intermediate layer of the machine learning model for the input data; and (2) a second loss between the ground truth data and an output of a final layer of the machine learning model, and wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
Claim 18 is a method claim that substantially corresponds to Claim 1, and Claim 20 is a computer medium claim that also substantially corresponds to Claim 1.
Claim 13 includes features similar to Claim 1 …
Claim 17 also includes features along the lines of Claim 1 …
Claim 19 is a method claim that substantially corresponds to Claim 17.
The applied art, whether taken alone or in any permissible combination, is not seen to disclose or to suggest at least the following features of independent Claims 1, 13 and 17 to 20: wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
Hachiya discloses that, to select recognition targets that match the user's specific domain and needs, and to streamline the pre-training and fine-tuning of the recognizer, the system calculates the relevance between the domain and candidate recognition targets based on an ontology that represents the conceptual structure of the domain. Hachiya then selects the appropriate recognition targets and trains the recognizer using training data for those targets. The domain refers to usage environments or scenes such as stores or train stations. However, Hachiya is not seen to disclose or to suggest at least the features of, wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
Li discloses that, to prevent performance degradation when applying the model to a new domain that differs from the domain of the training data, the system extracts feature representations of the new-domain data using an existing model, generates probability distributions for each region, and updates the parameters using the loss calculated by comparing these distributions with the ground truth (GT) of the new domain. The domain refers to attribute differences across the entire dataset. However, Li is not seen to disclose or to suggest anything that, when combined with Hachiya, would have resulted in the features of, wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
Kiraly is also not seen to disclose or to suggest anything that, when combined with Hachiya and/or Li, would have resulted in the above features of Claims 1, 13 and 17 to 20. In this regard, Kiraly is seen to disclose that ground truth response maps are generated by creating Gaussian distributions centered at the tumor locations based on the annotated ground truth tumor locations and classifications. The response maps are merely overlaid using different colors for visualization purposes and are not generated based on color. However, Kiraly is not seen to disclose or to suggest anything that, when combined with Hachiya and/or Li, would have resulted in the features of, wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
In view of the foregoing, amended independent Claims 1, 13 and 17 to 20 are believed to be allowable over the proposed combination of Hachiya, Li and Kiraly, and reconsideration and withdrawal of the rejections are respectfully requested.
Castillo, Kuroda and Zoph have been studied, but none of these references are seen to make up for the deficiencies of Hachiya, Li and Kiraly. In this regard, Castillo is seen to disclose to prevent degradation in the quality of downstream processing in computer vision (CV), the system generates an aggregated mask by voting and weighting masks across multiple frames. Kuroda is seen to disclose to prevent incorrect learning in multi-label training and similar tasks, an evaluation mechanism for excluding labels is introduced. Zoph is seen to disclose an RNN controller which generates a neural network architecture, learns and evaluates the model, and trains RNN using obtained accuracies as rewards. However, none of Castillo, Kuroda or Zoph are seen to disclose or to suggest anything that, when combined with Hachiya, Li and/or Kiraly, would have resulted in the features of, wherein the specific domain ground truth data is generated from (a) a response map of the input data generated, by a model that responds to a specific color of the input data, and (b) the ground truth data.
In view of the foregoing amendments and remarks, all of Claims 1 to 22 are believed to be allowable, and reconsideration and withdrawal of the rejections are requested” (Remarks Pages 12-17).
Examiner’s Response:
The examiner respectfully disagrees. Regarding claim 1, the examiner respectfully disagrees to applicant’s assertion that “wherein the specific domain ground truth data is generated from (a) a response map of the input data …, and (b) the ground truth data” is at least not disclosed by the cited prior arts. In particular, examiner points to paragraphs [0027]-[0029] of Kiraly et al. (US 2018/0240233 A1), which specifically disclose, with respect to Fig. 1, “Two ground truth response maps corresponding to benign and malignant tumor labels, respectively, are created for each training mpMRI image set based on the annotated ground truth tumor locations and classifications. For a given training mpMRI image set, the ground truth response map for a given tumor class (e.g., malignant or benign) is created by generating a Gaussian distribution (e.g., with 3σ of 10 mm) centered at each ground truth tumor center location of the given class in a 2D slice containing the ground truth tumor center location … the image-to-image convolutional encoder-decoder can be trained to input a 2D slice from each of a plurality mpMRI images (e.g., T2-weighted, ADC, high b-value DWI, and K-Trans) and generate/output a respective 2D response map corresponding to the input slice for each of the tumor classes (e.g., benign and malignant). In this case, a received 3D mpMRI image set of a patient can be input to the trained image-to-image convolutional encoder-decoder slice by slice (with each input set of images including a corresponding slice from each of mpMRI images), and the trained image-to-image convolutional encoder-decoder can perform simultaneous detection and classification of prostate tumors for each input slice and generate 2D benign and malignant tumor response maps corresponding to each input slice ... the prostate tumor detection and classification results can be output by overlaying the generated benign tumor response maps and malignant tumor response maps on structural MRI images, such as T2-weighted MRI images. For example, 2D response maps generated for benign and malignant tumors can be overlaid on corresponding slices of the T2-weighted MRI image of the patient and displayed on a display device. In a possible embodiment, the generated benign tumor response maps and the generated malignant tumor response maps can be overlaid on slices of the T2- weighted MRI image using different colors to represent detected benign and malignant tumors in order to provide an easily distinguishable indication of the classification of each detected tumor” (i.e. teaches generating ground truth response maps (specific domain ground truth data) for a specific domain (e.g. in this example, benign and malignant tumors are different specific domains) based on the generated classifications in a slice (response map) and the ground truth data and that the generated classifications in a slice are response maps that are corresponding to a specific domain/classification). The ground truth response maps for a specific domain (specific domain ground truth data) are generated based on a response map generated for the specific domain along with ground truth data.
Applicant’s other arguments, filed 07/31/2026, with respect to the 35 U.S.C. 103 prior art rejections, including the arguments regarding the “a response map of the input data generated, by a model that responds to a specific color of the input data” limitation of claim 1, have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant relies on the arguments above regarding independent claims 13 and 17-20 and dependent claims 2-12, 14-16, and 21-22 therefore the response above is applicable to those claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIAN J HALES whose telephone number is (571)272-0878. The examiner can normally be reached M-F 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRIAN J HALES/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights