DETAILED ACTION
This Action is in response to Applicant’s response filed on 04/17/2026. Claims 1-20 are still pending in the present application. This Action is made FINAL.
Response to Amendment
Claim Objection: The amended claims filed on 04/17/2026overcomes the Claim Objection in the previous office action.
Claim 35 U.S.C. 112(b) Rejection: The amended claims filed on 04/17/2026 overcomes the Claim 112(b) Rejection in the previous office action.
Response to Arguments
Applicant's arguments filed on 04/17/2026 have been fully considered but are moot in view of the new ground(s) rejection in view of Li (U.S. 20180365834 A1)
Claim Status
Claim(s) 1-3, 5, 7-14 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishida et al (WO-2021171732 A1; Nishida), in view of Li (U.S. 20180365834 A1).
Claim(s) 4, 6 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishida et al (WO-2021171732 A1; Nishida),in view of Li (U.S. 20180365834 A1), and in further view of MA (U.S. 20220300706 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 5, 7-14 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishida et al (WO-2021171732 A1; Nishida), in view of Li (U.S. 20180365834 A1).
Regarding claim 1, Nishida discloses a method of generating a language feature extraction model that causes a computer to execute processing of extracting a feature from a text related to an image, (Paragraph 5: “a learning device, a text generation device, a learning method, a text generation method, and a program.”; Paragraph 13: “the question answering device 10 according to the present embodiment is not only the position and size of the text in the image, but also visual information such as a graph and a photograph included in the image (in other words, an aid to help understanding the text).” ) the method comprising: by a system including one or more processors, (Paragraph 220: “ Each functional unit included in the question answering device 10 is implemented, for example, through processing that the one or more programs stored in the memory device 206 causes the processor 205 to execute.”)
with performing of machine learning using a plurality of pieces of training data including a first image, ((Figs. 1 and 10: Image including text) first position information related to a region of interest in the first image, (Figs. 10: set of correct feature regions); and a first text (Figs. 1 and 10: question text related to image) that describes the region of interest (Paragraph 17; Paragraph 121: “It is assumed that training data input into a question answering device 10 in the learning time includes a set of correct feature regions, in addition to an image including text, a question text, and a correct answer. The set of correct feature regions is a set of feature regions necessary to obtain the correct answer, among feature regions extracted from the image.”) to
input the first text (Figs. 1 and 10: question text related to image) into a first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104) to cause the first model to output a first feature amount representing a feature of the first text, (Paragraphs 22-23: “The text analysis unit 103 divides each of the text output from the text recognition unit 102 and an input question text into a sequence of tokens … the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration. In other words, language understanding can be achieved that also takes a visual effect in the image into consideration.”)
input the first image feature and the first feature amount (Figs. 1 and 10: Text analysis unit: output a sequence of tokens from input question text; Language with visual effect understand unit 104: coded sequence in which visual information is taken into consideration is obtained) into a second model (Figs. 1 and 10: related feature region determination unit 108) different from the first model to cause the second model to estimate the region of interest in the first image, (Paragraphs 125: “The related feature region determination unit 108 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, calculates a probability indicating whether or not a feature region extracted by the feature region extraction unit 101 is information necessary to answer a question.”; Paragraphs 154-155: “the language understanding unit 104 with visual effects converts the coded sequence H obtained in step S802 above into a vector sequence H' by the Transformer Encoder of the M layer (step S803). That is, the language understanding unit 104with visual effects sets H'= TransformerEncoder (H) … the related feature area determination unit 108 calculates a probability indicating whether or not the feature area is a region necessary for answer generation (step S804). That is, if the element of H' corresponding to the subword token x (however, the area token or the document token) in the input token series is h', the related feature area determination unit 108 corresponds to the subword token x. The probability that the characteristic area to be used is necessary for the correct answer is calculated”) and
train the first model and the second model such that an estimated region of interest output from the second model matches the region of interest of a correct answer indicated by the first position information, (Paragraphs 125-126: “ the learning model parameters stored in the parameter storage unit 107 also include the learning model parameters of the neural network model that realizes the related feature region determination unit 108. The parameter learning unit 106 calculates a loss by using also the probability calculated by the related feature region determination unit 108 and the set of correct feature regions, and updates the model parameters being learned that are stored in the parameter storage unit 107.”; Paragraph 182: “the language understanding unit 104 with visual effects, the answer text generation unit 105, and the related feature area determination unit 108 use the learned model parameters stored in the parameter storage unit 107 … The related feature area determination unit 108 is calculated or determined from the probability indicating whether or not the feature area extracted by the feature area extraction unit 101 is information necessary for answering a question ( Related feature area score) may be output.)
generating the first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104), which is the language feature extraction model. (Paragraph 22-25: “the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration … The parameter storage unit 107 stores the model parameters being learned (that is, model parameters to be learned) of the neural network models that implement the language-with-visual-effect understanding unit 104 and the answer text generation unit 105.”)
However, Nishida does not disclose input the first image and the first feature amount into a second model different from the first model to cause the second model to estimate generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount,
Li discloses performing of machine learning using a plurality of pieces of training data including a first image, (a plurality of pieces of image data) first position information related to a region of interest in the first image (an interpretation report), (Paragraph 9: “a learning data generation support apparatus including storage means for storing a plurality of pieces of image data and an interpretation report with respect to each of the plurality of pieces of image data”; Paragraph 61: “The learning means 15 performs machine learning based on a neural network using the image data registered in the registration unit 16 as the correct answer data by the registration means 14. Specifically, for example, by using a convolutional neural network, or the like, an image recognition device may be generated.”) and
a first text (Fig.3: “an interpretation report”) that describes the region of interest to input the first text into a first model (Fig.2 the extraction means 11) to cause the first model (Fig.2 the extraction means 11) to output a first feature amount representing a feature of the first text (“lesion name”, “lesion portion position information”, “size”, and “tumor feature”,) (Fig.3 and Paragraph 49-50: “ interpretation report of a patient A of a lung disease is extracted from an interpretation report database 8 (S1) … natural language processing is executed with respect to the interpretation report by the extraction means 11 (S2), to thereby extract character strings corresponding to “lesion name”, “lesion portion position information”, “size”, and “tumor feature”, respectively (S3).”)
input the first image (Fig.3: “image data and Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4).”) and the first feature amount (Fig.3 and Paragraph 51: “the lesion name extracted from the interpretation report is “pulmonary nodule””) into a second model (Fig.2 the analysis means 12) different from the first model (Fig.2 the extraction means 11) to cause the second model (Fig.2 the analysis means 12) to generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount, (Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4). Since the lesion name extracted from the interpretation report is “pulmonary nodule”, the analysis means 12 performs a lung field recognition process with respect to the CT image (S5). Since the lesion portion position information in the interpretation report indicates “right upper lobe”, the analysis means 12 performs a lesion analysis process for a pulmonary nodule in the extracted lung field region (S6). … Further, the size of the detected abnormal shadow is measured, and then, the size and the tumor feature are extracted as a lesion feature (the second lesion feature) (S7).”)
train the second model such that an estimated region of interest output from the second model matches the region of interest indicated by the first position information (Fig. 3 and Paragraph 54: “in a case where the size of the tumor recorded in the interpretation report is two or more times larger than the size of the tumor extracted by the analysis means 12, it cannot be determined that the tumors are the same (NO in S8 and NO in S10), a parameter for an image analysis process, particularly, a parameter relating to a lesion analysis process for various lesions is adjusted (S11), and then, the lesion analysis process that is the image analysis process is performed again (S6). … the procedure returns to the organ recognition of S5 after the parameter is adjusted, but the procedure may return to the lesion analysis process of S6 according to a condition in which it is determined that the matching is not achieved.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Nishida by including an image analysis process that is taught by Li, to make the invention a learning data generation support program that perform support for generating learning data through machine learning; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving recognize features of images with high accuracy using deep learning as well as automatically acquire a large amount of various correct answer data necessary for deep learning.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 2, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses comprising: by the system, using a third model (Fig. 10: answer text generation unit) that receives inputs of an image feature amount extracted from the image and a language feature amount extracted from the text and outputs a degree of association between the two feature amounts; (Paragraphs 23-24: “The answer text generation unit 105 is realized by a neural network, and the generation probability of the answer text is calculated from the coded sequence obtained by the language understanding unit 104 with visual effects by using the learning model parameters stored in the parameter storage unit 107. Calculate the probability distribution to represent. …the parameter learning unit 106 updates the learning model parameters stored in the parameter storage unit 107 by using the loss between the answer text generated by the answer text generation unit 105 and the input correct answer text.”, it shows that the answer text generated from image tokens read as “second feature amount” and sequences of text token read as “first feature amount”)
in the machine learning, inputting of a second feature amount extracted from the first image (Paragraph 39: “FIG. 4 shows an example of the extraction of feature regions by the feature region extraction unit 101. In the example shown in FIG. 4, a case is illustrated in which five feature regions including a feature region 1100, a feature region 1200, a feature region 1300, a feature region 1400, and a feature region 1500 are extracted from an image 1000 including text.”) and the first feature amount into the third model (Fig. 10: answer text generation unit) to cause the third model to estimate a degree of association between the first image and the first text; (Paragraph 23: “The answer text generation unit 105 is realized by a neural network, and the generation probability of the answer text is calculated from the coded sequence obtained by the language understanding unit 104 with visual effects by using the learning model parameters stored in the parameter storage unit 107. Calculate the probability distribution to represent.”, it shows that the answer text generated from image tokens read as “second feature amount” and sequences of text token read as “first feature amount”), and training of the first model and the third model such that an estimated degree of association output from the third model matches a degree of association of the correct answer.(Paragraph 24: “the parameter learning unit 106 updates the learning model parameters stored in the parameter storage unit 107 by using the loss between the answer text generated by the answer text generation unit 105 and the input correct answer text.”)
Regarding claim 3, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses comprising by the system, using a fourth model (Figs. 1 and 10: feature region extraction unit 101) that extracts the second feature amount from the input first image, (Paragraph 39: “in FIG. 4, a case is illustrated in which five feature regions including a feature region 1100, a feature region 1200, a feature region 1300, a feature region 1400, and a feature region 1500 are extracted from an image 1000 including text.”)
in the machine learning, inputting of the first image and the first position information into the fourth model to cause the fourth model to output the second feature amount, (Paragraph 39: “in FIG. 4, a case is illustrated in which five feature regions including a feature region 1100, a feature region 1200, a feature region 1300, a feature region 1400, and a feature region 1500 are extracted from an image 1000 including text.”; Paragraphs 132-133: “the feature region extraction unit 101 extracts K feature regions from an image included in the read training data (step S702). … for the location information, any information may be used as long as the information can specify a location of the feature region, …these nine types of regions are examples, and other region types may be set… at least two types are set, that is, an area type indicating that the feature area does not contain text and an area type indicating that the feature area contains text”) and
training of the first model, the third model, and the fourth model such that the estimated degree of association output from the third model matches the degree of association of the correct answer. (Paragraphs 23-24: “The answer text generation unit 105 is realized by a neural network, and the generation probability of the answer text is calculated from the coded sequence obtained by the language understanding unit 104 with visual effects by using the learning model parameters stored in the parameter storage unit 107. Calculate the probability distribution to represent. …the parameter learning unit 106 updates the learning model parameters stored in the parameter storage unit 107 by using the loss between the answer text generated by the answer text generation unit 105 and the input correct answer text.”, it shows that the answer text generated from image tokens read as “second feature amount” and sequences of text token read as “first feature amount”)
Regarding claim 5, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses the text and the first text are structured texts. (Paragraph 210: “a model that generates an answer to a question sentence by inputting a question sentence, a feature area, and a token of an OCR token”; Paragraph 121: “It is assumed that training data input into a question answering device 10 in the learning time includes a set of correct feature regions, in addition to an image including text, a question text, and a correct answer. The set of correct feature regions is a set of feature regions necessary to obtain the correct answer, among feature regions extracted from the image.”)
Regarding claim 7, Nishida, as modified by Li, discloses all the claims invention . Li further discloses comprising by the system, performing of processing of displaying the region of interest estimated by the second model. (Paragraph 27: “As the display device, one or plural high definition displays are provided. In the radiologist workstation 3, respective processes such as transmission request of an image with respect to the image management server 5, display of an image received from the image management server 5, automatic detection and highlighting of a lesion likeliness portion in an image, and creation and display of an interpretation report, and the like are performed by executing a software program for the respective processes”; Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4). Since the lesion name extracted from the interpretation report is “pulmonary nodule”, the analysis means 12 performs a lung field recognition process with respect to the CT image (S5).”)
Regarding claim 8, Nishida, as modified by Li, discloses all the claims invention. Li further discloses the first position information includes coordinate information that specifies a position of the region of interest in the first image. (Paragraphs 40: “The organ information (or tissue information) may be information from which an organ (or tissue) can be estimated, such as a name of the organ (or tissue) or a lesion name recorded in an interpretation report. The lesion portion position information includes coordinates of a central position of a lesion portion (for example, barycentric coordinates), the position or a range of a lesion in an organ (or tissue) (for example, a right upper lobe in the case of the lung, a lymph node #13R in the case of the lymph gland), a range of slices in which a lesion portion is present in the case of a three-dimensional image formed by a plurality of tomographic images, or the like.”)
Regarding claim 9, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses the first image is a cropped image including the first position information. (Paragraph 37: “the feature region extraction unit 101 extracts K feature regions from the image included in the read training data (step S202). The feature area is an area based on visual features, and is represented by a rectangular area in the present embodiment. The k-th feature area includes position information (7 dimensions in total) including upper left coordinates, lower right coordinates, width, height, and area, a rectangular image representation (D dimension), and an area type (C type).”)
Regarding claim 10, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses an information processing apparatus (Paragraph 5: “a learning device, a text generation device, a learning method, a text generation method, and a program.”) comprising: one or more storage devices that store a program including the language feature extraction model generated by the method of generating a language feature extraction model according to claim 1; and one or more processors that execute the program. (Paragraph 220: “ Each functional unit included in the question answering device 10 is implemented, for example, through processing that the one or more programs stored in the memory device 206 causes the processor 205 to execute.”)
Regarding claim 11, Nishida discloses an information processing apparatus (Paragraph 5: “a learning device, a text generation device, a learning method, a text generation method, and a program.” comprising: one or more processors; and one or more storage devices that store a command executed by the one or more processors, wherein the one or more processors (Paragraph 220: “ Each functional unit included in the question answering device 10 is implemented, for example, through processing that the one or more programs stored in the memory device 206 causes the processor 205 to execute.”) are configured to:
acquire a text that describes a region of interest in an image; (Paragraph 17: “set of training data (training data set) including an image including text, a question text related to this image, and a correct answer text indicating a correct answer to this question text is input to the question answering device 10 at the time of learning”) and
execute processing of inputting the text into a first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104) to cause the first model to output a language feature amount representing a feature of the text, (Paragraphs 22-23: “The text analysis unit 103 divides each of the text output from the text recognition unit 102 and an input question text into a sequence of tokens … the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration. In other words, language understanding can be achieved that also takes a visual effect in the image into consideration.”) and
the first model is a model obtained by performing machine learning using a plurality of pieces of training data including a first image (Figs. 1 and 10: Image including text) for training, first position information related to a region of interest in the first image, (Figs. 10: set of correct feature regions), and a first text that describes the region of interest (Figs. 1 and 10: question text related to image) ; (Paragraph 121: “It is assumed that training data input into a question answering device 10 in the learning time includes a set of correct feature regions, in addition to an image including text, a question text, and a correct answer. The set of correct feature regions is a set of feature regions necessary to obtain the correct answer, among feature regions extracted from the image.”) to input the first text (Figs. 1 and 10: question text related to image) into the first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104) to cause the first model to output a first feature amount representing a feature of the first text (Paragraphs 22-23: “The text analysis unit 103 divides each of the text output from the text recognition unit 102 and an input question text into a sequence of tokens … the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration. In other words, language understanding can be achieved that also takes a visual effect in the image into consideration.”)and
inputting of the first image feature and the first feature amount (Figs. 1 and 10: Text analysis unit output a sequence of tokens from input question text; Language with visual effect understand unit 104: coded sequence in which visual information is taken into consideration is obtained) into a second model (Fig. 10: The related feature region determination unit 108) different from the first model to cause the second model to estimate the region of interest in the first image, (Paragraphs 125: “The related feature region determination unit 108 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, calculates a probability indicating whether or not a feature region extracted by the feature region extraction unit 101 is information necessary to answer a question.”; Paragraphs 154-155: “the language understanding unit 104 with visual effects converts the coded sequence H obtained in step S802 above into a vector sequence H'by the Transformer Encoder of the M layer (step S803). That is, the language understanding unit 104with visual effects sets H'= TransformerEncoder (H) … the related feature area determination unit 108 calculates a probability indicating whether or not the feature area is a region necessary for answer generation (step S804). That is, if the element of H'corresponding to the subword token x (however, the area token or the document token) in the input token series is h', the related feature area determination unit 108 corresponds to the subword token x. The probability that the characteristic area to be used is necessary for the correct answer is calculated”) and
train the first model and the second model such that an estimated region of interest output from the second model matches the region of interest of a correct answer indicated by the first position information. (Paragraphs 125-126: “ the learning model parameters stored in the parameter storage unit 107 also include the learning model parameters of the neural network model that realizes the related feature region determination unit 108. The parameter learning unit 106 calculates a loss by using also the probability calculated by the related feature region determination unit 108 and the set of correct feature regions, and updates the model parameters being learned that are stored in the parameter storage unit 107.”; Paragraph 182: “the language understanding unit 104 with visual effects, the answer text generation unit 105, and the related feature area determination unit 108 use the learned model parameters stored in the parameter storage unit 107 … The related feature area determination unit 108 is calculated or determined from the probability indicating whether or not the feature area extracted by the feature area extraction unit 101 is information necessary for answering a question ( Related feature area score) may be output.)
However, Nishida does not disclose inputting the first image and the first feature amount into a second model different from the first model to cause the second model to estimate generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount,
Li discloses performing machine learning using a plurality of pieces of training data including a first image for training, (a plurality of pieces of image data) first position information related to a region of interest in the first image (an interpretation report), (Paragraph 9: “a learning data generation support apparatus including storage means for storing a plurality of pieces of image data and an interpretation report with respect to each of the plurality of pieces of image data”; Paragraph 61: “The learning means 15 performs machine learning based on a neural network using the image data registered in the registration unit 16 as the correct answer data by the registration means 14. Specifically, for example, by using a convolutional neural network, or the like, an image recognition device may be generated.”) and
a first text (Fig.3: “an interpretation report”) that describes the region of interest to input the first text into a first model (Fig.2 the extraction means 11) to cause the first model (Fig.2 the extraction means 11) to output a first feature amount representing a feature of the first text (“lesion name”, “lesion portion position information”, “size”, and “tumor feature”,) (Fig.3 and Paragraph 49-50: “ interpretation report of a patient A of a lung disease is extracted from an interpretation report database 8 (S1) … natural language processing is executed with respect to the interpretation report by the extraction means 11 (S2), to thereby extract character strings corresponding to “lesion name”, “lesion portion position information”, “size”, and “tumor feature”, respectively (S3).”) and
inputting the first image (Fig.3: “image data and Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4).”) and the first feature amount (Fig.3 and Paragraph 51: “the lesion name extracted from the interpretation report is “pulmonary nodule””) into a second model (Fig.2 the analysis means 12) different from the first model (Fig.2 the extraction means 11) to cause the second model (Fig.2 the analysis means 12) to generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount, (Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4). Since the lesion name extracted from the interpretation report is “pulmonary nodule”, the analysis means 12 performs a lung field recognition process with respect to the CT image (S5). Since the lesion portion position information in the interpretation report indicates “right upper lobe”, the analysis means 12 performs a lesion analysis process for a pulmonary nodule in the extracted lung field region (S6). … Further, the size of the detected abnormal shadow is measured, and then, the size and the tumor feature are extracted as a lesion feature (the second lesion feature) (S7).”)
train the second model such that an estimated region of interest output from the second model matches the region of interest indicated by the first position information (Fig. 3 and Paragraph 54: “in a case where the size of the tumor recorded in the interpretation report is two or more times larger than the size of the tumor extracted by the analysis means 12, it cannot be determined that the tumors are the same (NO in S8 and NO in S10), a parameter for an image analysis process, particularly, a parameter relating to a lesion analysis process for various lesions is adjusted (S11), and then, the lesion analysis process that is the image analysis process is performed again (S6). … the procedure returns to the organ recognition of S5 after the parameter is adjusted, but the procedure may return to the lesion analysis process of S6 according to a condition in which it is determined that the matching is not achieved.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Nishida by including an image analysis process that is taught by Li, to make the invention a learning data generation support program that perform support for generating learning data through machine learning; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving recognize features of images with high accuracy using deep learning as well as automatically acquire a large amount of various correct answer data necessary for deep learning.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 12, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses the one or more processors are configured to: input an image feature amount extracted from a second image (Paragraph 39: “in FIG. 4, a case is illustrated in which five feature regions including a feature region 1100, a feature region 1200, a feature region 1300, a feature region 1400, and a feature region 1500 are extracted from an image 1000 including text.”; Paragraph 143: “it is assumed that the token corresponding to the area type with the k-th feature region. For example, if the area type of the kth feature area is "image"”), and a language feature amount extracted from the text (The text analysis unit 103 divides the input question text into token series) into a third model to cause the third model to output a degree of association between the second image and the text. (Paragraphs 21-23: “The text analysis unit 103 divides the text output by the text recognition unit 102 and the input question text into token series, respectively. … The answer text generation unit 105 is realized by a neural network, and the generation probability of the answer text is calculated from the coded sequence obtained by the language understanding unit 104 with visual effects by using the learning model parameters stored in the parameter storage unit 107. Calculate the probability distribution to represent.”, it shows that the answer text generated from image tokens which is k-regions read as “feature amount extract from a second image” and sequences of text token read as “ a language feature amount extract from text”)
Regarding claim 13, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses the one or more processors are configured to: input an image feature amount extracted from a second image (Paragraph 39: “in FIG. 4, a case is illustrated in which five feature regions including a feature region 1100, a feature region 1200, a feature region 1300, a feature region 1400, and a feature region 1500 are extracted from an image 1000 including text.”; Paragraph 143: “it is assumed that the token corresponding to the area type with the k-th feature region. For example, if the area type of the kth feature area is "image"”), and the language feature amount extracted from the text (The text analysis unit 103 divides the input question text into token series) into a third model to cause the third model to output a degree of association between the second image and the text. (Paragraphs 21-23: “The text analysis unit 103 divides the text output by the text recognition unit 102 and the input question text into token series, respectively. … The answer text generation unit 105 is realized by a neural network, and the generation probability of the answer text is calculated from the coded sequence obtained by the language understanding unit 104 with visual effects by using the learning model parameters stored in the parameter storage unit 107. Calculate the probability distribution to represent.”, it shows that the answer text generated from image tokens which is k-regions read as “feature amount extract from a second image” and sequences of text token read as “ a language feature amount extract from text”)
Regarding claim 14, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses the one or more processors are configured to: acquire the second image and second position information related to a region of interest in the second image; (Paragraph 39: “in FIG. 4, a case is illustrated in which five feature regions including a feature region 1100, a feature region 1200, a feature region 1300, a feature region 1400, and a feature region 1500 are extracted from an image 1000 including text.”; Paragraph 143: “it is assumed that the token corresponding to the area type with the k-th feature region. For example, if the area type of the kth feature area is "image"”), and input the second image and the second position information into a fourth model (Figs.1 and 10: the feature region extraction unit 101) to cause the fourth model to output the image feature amount. (Paragraph 136; Paragraph 37: “the feature region extraction unit 101 extracts K feature regions from the image included in the read training data (step S202). The feature area is an area based on visual features, and is represented by a rectangular area. The k-th feature area includes position information (7 dimensions in total) including upper left coordinates, lower right coordinates, width, height, and area, a rectangular image representation (D dimension), and an area type (C type). It shall be represented as an image token i .sup.k with. However, as the position information, any information may be used as long as the position of the feature area can be specified (for example, at least one information of width, height and area may not be used, and the upper left coordinates and the lower right may be used. Instead of the coordinates, the upper right and lower left coordinates may be used, or the center coordinates may be used).”, it shows that the feature extraction unit extract K feature regions read as “plurality of image or second image” and include position information such as upper and/or lower left/right coordinate is interpreted as “plurality position information and/or second position information);.
Regarding claim 17, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses the text and the first text are structured texts. (Paragraph 210: “a model that generates an answer to a question sentence by inputting a question sentence, a feature area, and a token of an OCR token”; Paragraph 121: “It is assumed that training data input into a question answering device 10 in the learning time includes a set of correct feature regions, in addition to an image including text, a question text, and a correct answer. The set of correct feature regions is a set of feature regions necessary to obtain the correct answer, among feature regions extracted from the image.”)
Regarding claim 18, Nishida, as modified by Li, discloses all the claims invention. Nishida further discloses the text and the first text are structured texts. (Paragraph 210: “a model that generates an answer to a question sentence by inputting a question sentence, a feature area, and a token of an OCR token”; Paragraph 121: “It is assumed that training data input into a question answering device 10 in the learning time includes a set of correct feature regions, in addition to an image including text, a question text, and a correct answer. The set of correct feature regions is a set of feature regions necessary to obtain the correct answer, among feature regions extracted from the image.”)
Regarding claim 19, Nishida discloses an information processing method (Paragraph 5: “a learning device, a text generation device, a learning method, a text generation method, and a program.” ) comprising: by one or more processors, (Paragraph 220: “ Each functional unit included in the question answering device 10 is implemented, for example, through processing that the one or more programs stored in the memory device 206 causes the processor 205 to execute.”)
acquiring a text that describes a region of interest in an image; (Paragraph 17: “set of training data (training data set) including an image including text, a question text related to this image, and a correct answer text indicating a correct answer to this question text is input to the question answering device 10 at the time of learning”) and
executing processing of inputting the text into a first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104) to cause the first model to output a language feature amount representing a feature of the text, (Paragraphs 22-23: “The text analysis unit 103 divides each of the text output from the text recognition unit 102 and an input question text into a sequence of tokens … the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration. In other words, language understanding can be achieved that also takes a visual effect in the image into consideration.”)
wherein the first model is a model obtained by performing machine learning using training data including a first image for training (Figs. 1 and 10: Image including text), a first text that describes a region of interest in the first image (Figs. 1 and 10: question text related to image), and first position information related to the region of interest in the first image (Figs. 10: set of correct feature regions) ; (Paragraph 121: “It is assumed that training data input into a question answering device 10 in the learning time includes a set of correct feature regions, in addition to an image including text, a question text, and a correct answer. The set of correct feature regions is a set of feature regions necessary to obtain the correct answer, among feature regions extracted from the image.”)
to input the first text (Figs. 1 and 10: question text related to image) into the first model to cause the first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104) to output a first feature amount representing a feature of the first text (Paragraphs 22-23: “The text analysis unit 103 divides each of the text output from the text recognition unit 102 and an input question text into a sequence of tokens … the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration. In other words, language understanding can be achieved that also takes a visual effect in the image into consideration.”) and
inputting of the first image feature and the first feature amount (Figs. 1 and 10: Text analysis unit output a sequence of tokens from input question text; Language with visual effect understand unit 104: coded sequence in which visual information is taken into consideration is obtained) into a second model (Fig. 10: The related feature region determination unit 108) different from the first model to cause the second model to estimate the region of interest in the first image, (Paragraphs 125: “The related feature region determination unit 108 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, calculates a probability indicating whether or not a feature region extracted by the feature region extraction unit 101 is information necessary to answer a question.”; Paragraphs 154-155: “the language understanding unit 104 with visual effects converts the coded sequence H obtained in step S802 above into a vector sequence H'by the Transformer Encoder of the M layer (step S803). That is, the language understanding unit 104with visual effects sets H'= TransformerEncoder (H) … the related feature area determination unit 108 calculates a probability indicating whether or not the feature area is a region necessary for answer generation (step S804). That is, if the element of H'corresponding to the subword token x (however, the area token or the document token) in the input token series is h', the related feature area determination unit 108 corresponds to the subword token x. The probability that the characteristic area to be used is necessary for the correct answer is calculated”) and
train the first model and the second model such that the region of interest estimated by the second model matches the region of interest indicated by the first position information. (Paragraphs 125-126: “ the learning model parameters stored in the parameter storage unit 107 also include the learning model parameters of the neural network model that realizes the related feature region determination unit 108. The parameter learning unit 106 calculates a loss by using also the probability calculated by the related feature region determination unit 108 and the set of correct feature regions, and updates the model parameters being learned that are stored in the parameter storage unit 107.”; Paragraph 182: “the language understanding unit 104 with visual effects, the answer text generation unit 105, and the related feature area determination unit 108 use the learned model parameters stored in the parameter storage unit 107 … The related feature area determination unit 108 is calculated or determined from the probability indicating whether or not the feature area extracted by the feature area extraction unit 101 is information necessary for answering a question ( Related feature area score) may be output.)
However, Nishida does not disclose inputting the first image and the first feature amount into a second model different from the first model to cause the second model to estimate generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount,
Li discloses performing machine learning using a plurality of pieces of training data including a first image for training, (a plurality of pieces of image data) a first text that describes a region of interest in the first image,(an interpretation report) and first position information (an interpretation report) related to the region of interest in the first image Paragraph 9: “a learning data generation support apparatus including storage means for storing a plurality of pieces of image data and an interpretation report with respect to each of the plurality of pieces of image data”; Paragraph 61: “The learning means 15 performs machine learning based on a neural network using the image data registered in the registration unit 16 as the correct answer data by the registration means 14. Specifically, for example, by using a convolutional neural network, or the like, an image recognition device may be generated.”) to input the first text (Fig.3: “an interpretation report”) into the first model (Fig.2 the extraction means 11) to cause the first model (Fig.2 the extraction means 11) to output a first feature amount representing a feature of the first text (Fig.3 and Paragraph 49-50: “ interpretation report of a patient A of a lung disease is extracted from an interpretation report database 8 (S1) … natural language processing is executed with respect to the interpretation report by the extraction means 11 (S2), to thereby extract character strings corresponding to “lesion name”, “lesion portion position information”, “size”, and “tumor feature”, respectively (S3).”) and
inputting the first image (Fig.3: “image data and Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4).”) and the first feature amount (Fig.3 and Paragraph 51: “the lesion name extracted from the interpretation report is “pulmonary nodule””) into a second model (Fig.2 the analysis means 12) different from the first model (Fig.2 the extraction means 11) to cause the second model (Fig.2 the analysis means 12) to generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount, (Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4). Since the lesion name extracted from the interpretation report is “pulmonary nodule”, the analysis means 12 performs a lung field recognition process with respect to the CT image (S5). Since the lesion portion position information in the interpretation report indicates “right upper lobe”, the analysis means 12 performs a lesion analysis process for a pulmonary nodule in the extracted lung field region (S6). … Further, the size of the detected abnormal shadow is measured, and then, the size and the tumor feature are extracted as a lesion feature (the second lesion feature) (S7).”)
train the second model such that an estimated region of interest output from the second model matches the region of interest indicated by the first position information (Fig. 3 and Paragraph 54: “in a case where the size of the tumor recorded in the interpretation report is two or more times larger than the size of the tumor extracted by the analysis means 12, it cannot be determined that the tumors are the same (NO in S8 and NO in S10), a parameter for an image analysis process, particularly, a parameter relating to a lesion analysis process for various lesions is adjusted (S11), and then, the lesion analysis process that is the image analysis process is performed again (S6). … the procedure returns to the organ recognition of S5 after the parameter is adjusted, but the procedure may return to the lesion analysis process of S6 according to a condition in which it is determined that the matching is not achieved.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Nishida by including an image analysis process that is taught by Li, to make the invention a learning data generation support program that perform support for generating learning data through machine learning; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving recognize features of images with high accuracy using deep learning as well as automatically acquire a large amount of various correct answer data necessary for deep learning.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 20, Nishida discloses a non-transitory, computer-readable tangible recording medium which records thereon a program that causes a computer to realize a function Paragraph 220: “ Each functional unit included in the question answering device 10 is implemented, for example, through processing that the one or more programs stored in the memory device 206 causes the processor 205 to execute.”) of extracting a feature from a text related to an image, (Paragraph 5: “a learning device, a text generation device, a learning method, a text generation method, and a program.”) the program causing the computer to realize:
a function of acquiring a text that describes a region of interest in the image; (Paragraph 17: “set of training data (training data set) including an image including text, a question text related to this image, and a correct answer text indicating a correct answer to this question text is input to the question answering device 10 at the time of learning”) and
a function of inputting the text into a first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104) to cause the first model to output a language feature amount representing a feature of the text, (Paragraphs 22-23: “The text analysis unit 103 divides each of the text output from the text recognition unit 102 and an input question text into a sequence of tokens … the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration. In other words, language understanding can be achieved that also takes a visual effect in the image into consideration.”)
wherein the first model is a model obtained by performing machine learning using training data including a first image for training (Figs. 1 and 10: Image including text), first position information related to a region of interest in the first image (Figs. 10: set of correct feature regions), and a first text that describes the region of interest in the first image (Figs. 1 and 10: question text related to image); (Paragraph 121: “It is assumed that training data input into a question answering device 10 in the learning time includes a set of correct feature regions, in addition to an image including text, a question text, and a correct answer. The set of correct feature regions is a set of feature regions necessary to obtain the correct answer, among feature regions extracted from the image.”)
to input the first text (Figs. 1 and 10: question text related to image) into the first model to cause the first model (Figs. 1 and 10: text analysis unit 103 and Language with visual effect understanding unit 104) to output a first feature amount representing a feature of the first text (Paragraphs 22-23: “The text analysis unit 103 divides each of the text output from the text recognition unit 102 and an input question text into a sequence of tokens … the language-with-visual-effect understanding unit 104 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, encodes sequences of tokens obtained by the text analysis unit 103. Thus, an encoded sequence can be obtained that takes visual information into consideration. In other words, language understanding can be achieved that also takes a visual effect in the image into consideration.”) and
inputting of the first image feature and the first feature amount (Figs. 1 and 10: Text analysis unit output a sequence of tokens from input question text; Language with visual effect understand unit 104: coded sequence in which visual information is taken into consideration is obtained) into a second model (Fig. 10: The related feature region determination unit 108) different from the first model to cause the second model to estimate the region of interest in the first image, (Paragraphs 125: “The related feature region determination unit 108 is implemented by a neural network and, by using model parameters being learned that are stored in the parameter storage unit 107, calculates a probability indicating whether or not a feature region extracted by the feature region extraction unit 101 is information necessary to answer a question.”; Paragraphs 154-155: “the language understanding unit 104 with visual effects converts the coded sequence H obtained in step S802 above into a vector sequence H'by the Transformer Encoder of the M layer (step S803). That is, the language understanding unit 104with visual effects sets H'= TransformerEncoder (H) … the related feature area determination unit 108 calculates a probability indicating whether or not the feature area is a region necessary for answer generation (step S804). That is, if the element of H'corresponding to the subword token x (however, the area token or the document token) in the input token series is h', the related feature area determination unit 108 corresponds to the subword token x. The probability that the characteristic area to be used is necessary for the correct answer is calculated”) and
train the first model and the second model such that an estimated region of interest output from the second model matches the region of interest indicated by the first position information. (Paragraphs 125-126: “ the learning model parameters stored in the parameter storage unit 107 also include the learning model parameters of the neural network model that realizes the related feature region determination unit 108. The parameter learning unit 106 calculates a loss by using also the probability calculated by the related feature region determination unit 108 and the set of correct feature regions, and updates the model parameters being learned that are stored in the parameter storage unit 107.”; Paragraph 182: “the language understanding unit 104 with visual effects, the answer text generation unit 105, and the related feature area determination unit 108 use the learned model parameters stored in the parameter storage unit 107 … The related feature area determination unit 108 is calculated or determined from the probability indicating whether or not the feature area extracted by the feature area extraction unit 101 is information necessary for answering a question ( Related feature area score) may be output.)
However, Nishida does not disclose inputting the first image and the first feature amount into a second model different from the first model to cause the second model to estimate generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount,
Li discloses performing machine learning using a plurality of pieces of training data including a first image for training, (a plurality of pieces of image data) first position information related to a region of interest in the first image (an interpretation report), (Paragraph 9: “a learning data generation support apparatus including storage means for storing a plurality of pieces of image data and an interpretation report with respect to each of the plurality of pieces of image data”; Paragraph 61: “The learning means 15 performs machine learning based on a neural network using the image data registered in the registration unit 16 as the correct answer data by the registration means 14. Specifically, for example, by using a convolutional neural network, or the like, an image recognition device may be generated.”) and
a first text (Fig.3: “an interpretation report”) that describes the region of interest to input the first text into a first model (Fig.2 the extraction means 11) to cause the first model (Fig.2 the extraction means 11) to output a first feature amount representing a feature of the first text (“lesion name”, “lesion portion position information”, “size”, and “tumor feature”,) (Fig.3 and Paragraph 49-50: “ interpretation report of a patient A of a lung disease is extracted from an interpretation report database 8 (S1) … natural language processing is executed with respect to the interpretation report by the extraction means 11 (S2), to thereby extract character strings corresponding to “lesion name”, “lesion portion position information”, “size”, and “tumor feature”, respectively (S3).”) and
inputting the first image (Fig.3: “image data and Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4).”) and the first feature amount (Fig.3 and Paragraph 51: “the lesion name extracted from the interpretation report is “pulmonary nodule””) into a second model (Fig.2 the analysis means 12) different from the first model (Fig.2 the extraction means 11) to cause the second model (Fig.2 the analysis means 12) to generate estimated region information specifying a location of the region of interest in the first image based on the first feature amount, (Paragraph 51: “a CT image is extracted from the image database 6 as image data (S4). Since the lesion name extracted from the interpretation report is “pulmonary nodule”, the analysis means 12 performs a lung field recognition process with respect to the CT image (S5). Since the lesion portion position information in the interpretation report indicates “right upper lobe”, the analysis means 12 performs a lesion analysis process for a pulmonary nodule in the extracted lung field region (S6). … Further, the size of the detected abnormal shadow is measured, and then, the size and the tumor feature are extracted as a lesion feature (the second lesion feature) (S7).”)
train the second model such that an estimated region of interest output from the second model matches the region of interest indicated by the first position information (Fig. 3 and Paragraph 54: “in a case where the size of the tumor recorded in the interpretation report is two or more times larger than the size of the tumor extracted by the analysis means 12, it cannot be determined that the tumors are the same (NO in S8 and NO in S10), a parameter for an image analysis process, particularly, a parameter relating to a lesion analysis process for various lesions is adjusted (S11), and then, the lesion analysis process that is the image analysis process is performed again (S6). … the procedure returns to the organ recognition of S5 after the parameter is adjusted, but the procedure may return to the lesion analysis process of S6 according to a condition in which it is determined that the matching is not achieved.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Nishida by including an image analysis process that is taught by Li, to make the invention a learning data generation support program that perform support for generating learning data through machine learning; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving recognize features of images with high accuracy using deep learning as well as automatically acquire a large amount of various correct answer data necessary for deep learning.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Claim(s) 4, 6 and 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishida et al (WO-2021171732 A1; Nishida),in view of Li (U.S. 20180365834 A1), and in further view of MA (U.S. 20220300706 A1).
Regarding claim 4, Nishida, as modified by Li, discloses all the claims invention except comprising by the system, using a fifth model that receives an input of a language feature amount extracted from each of a plurality of texts and outputs a degree of association between the plurality of the texts, in the machine learning, inputting of a third feature amount, which is extracted, by the first model from a second text different from the first text by inputting the second text into the first model, and the first feature amount into the fifth model to cause the fifth model to estimate a degree of association between the first text and the second text, and training of the first model and the fifth model such that an estimated degree of association output from the fifth model matches a degree of association of a correct answer.
Ma discloses comprising by the system, using a fifth model (Fig.1: PS model training unit 102) that receives an input of a language feature amount extracted from each of a plurality of texts and outputs a degree of association between the plurality of the texts, (Paragraph 30: “The input text is a text sentence described in a natural language. For example, the training dataset creation unit 101 may create input text by randomly extracting a passage having a length equal to or longer than a predetermined length from a known unlabeled corpus (for example, document database or long text sentence).” ; Paragraph 55: “he PS model training unit 102 uses the first training dataset to train the PS model. Input text of the first training dataset may be referred to as first input data. Furthermore, an application result of the first training dataset may be referred to as second input data.”) in the machine learning, inputting of a third feature amount (Fig.3: second input data), which is extracted, by the first model (Fig.1: training dataset creation unit 101) from a second text different from the first text by inputting the second text into the first model, (Paragraph 30: “The input text is a text sentence described in a natural language. For example, the training dataset creation unit 101 may create input text by randomly extracting a passage having a length equal to or longer than a predetermined length from a known unlabeled corpus (for example, document database or long text sentence).” and the first feature amount (Fig. 3: first input data) into the fifth model to cause the fifth model to estimate a degree of association between the first text and the second text, (Paragraphs 56-58: “he PS model training unit 102 performs training (machine learning) on the PS model by using the input text (first input data) and the execution result (second input data) of the first training dataset as training data …. The PS model training unit 102 may optimize parameters by updating a parameter of tensor decomposition and a parameter of a neural network in a direction for decreasing a loss function that defines an error between the inference result of the machine learning model, with respect to the training data, and correct answer data, for example, using a gradient descent method.”; Paragraph 67: “an example is given in which the PS model is trained using a training dataset that combines the above input text “ . . . Lassen county had a population of 34,895. The racial makeup of Lassen county was 25,532 (73.2%) white (U.S. census), 2,834 (8.1%) African American (U.S. census) . . . ”, the application instruction sequence “DIFF (9, SUM (10, 12))”, and the application result “6529”) and training of the first model and the fifth model such that an estimated degree of association output from the fifth model matches a degree of association of a correct answer. (Paragraphs 57-58: “PS model training unit 102 performs reinforcement learning (reinforcement training) so that the instruction sequence estimated by the PS model approaches the application instruction sequence (correct answer data). … The PS model training unit 102 may optimize parameters by updating a parameter of tensor decomposition and a parameter of a neural network in a direction for decreasing a loss function that defines an error between the inference result of the machine learning model, with respect to the training data, and correct answer data, for example, using a gradient descent method.”; Paragraph 69: “The PS model training unit 102 performs training based on a similarity between instruction sequences on the basis of the output “SUM (9, DIFF (11, 12))” and the correct answer data “DIFF (9, SUM (10, 12))” of the PS model.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Nishida by including a program synthesis (PS) model training unit that is taught by MA, to make the invention that generates a natural language processing (NLP) model that is a machine learning model that executes processing on a document written in a natural language; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving a natural language processing model using a neural network as well as enhancing the program output from the NLP model.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 6, Nishida, as modified by Li and Ma, discloses all the claims invention. Ma further discloses the second text is a structured text. (Fig. 8 and Paragraph 146: “example has been given in which an English sentence is used as the text sentence. However, the embodiment is not limited to this, and may be applied to languages other than English, and may be variously modified and implemented.”)
Regarding claim 15, Nishida, as modified by Li, discloses all the claims invention except the one or more processors are configured to: input a language feature amount extracted from each of a plurality of texts by the first model into a fifth model to cause the fifth model to output a degree of association between the plurality of the texts.
Ma discloses input a language feature amount extracted from each of a plurality of texts by the first model (Fig.1: training dataset creation unit 101) (Paragraph 30: “The input text is a text sentence described in a natural language. For example, the training dataset creation unit 101 may create input text by randomly extracting a passage having a length equal to or longer than a predetermined length from a known unlabeled corpus (for example, document database or long text sentence).”) into a fifth model (Fig. 1 and 3: PS model training unit 102) to cause the fifth model to output a degree of association between the plurality of the texts. (Paragraph 55: “the PS model training unit 102 uses the first training dataset to train the PS model. Input text of the first training dataset may be referred to as first input data. Furthermore, an application result of the first training dataset may be referred to as second input data.” Paragraph 67: “an example is given in which the PS model is trained using a training dataset that combines the above input text “ . . . Lassen county had a population of 34,895. The racial makeup of Lassen county was 25,532 (73.2%) white (U.S. census), 2,834 (8.1%) African American (U.S. census) . . . ”, the application instruction sequence “DIFF (9, SUM (10, 12))”, and the application result “6529”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Nishida by including a program synthesis (PS) model training unit that is taught by MA, to make the invention that generates a natural language processing (NLP) model that is a machine learning model that executes processing on a document written in a natural language; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving a natural language processing model using a neural network as well as enhancing the program output from the NLP model.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 16, Nishida, as modified by Li, discloses all the claims invention except the one or more processors are configured to: input a language feature amount extracted from each of a plurality of texts by the first model into a fifth model to cause the fifth model to output a degree of association between the plurality of the texts.
Ma discloses input a language feature amount extracted from each of a plurality of texts by the first model (Fig.1: training dataset creation unit 101) (Paragraph 30: “The input text is a text sentence described in a natural language. For example, the training dataset creation unit 101 may create input text by randomly extracting a passage having a length equal to or longer than a predetermined length from a known unlabeled corpus (for example, document database or long text sentence).”) into a fifth model (Fig. 1 and 3: PS model training unit 102) to cause the fifth model to output a degree of association between the plurality of the texts. (Paragraph 55: “the PS model training unit 102 uses the first training dataset to train the PS model. Input text of the first training dataset may be referred to as first input data. Furthermore, an application result of the first training dataset may be referred to as second input data.” Paragraph 67: “an example is given in which the PS model is trained using a training dataset that combines the above input text “ . . . Lassen county had a population of 34,895. The racial makeup of Lassen county was 25,532 (73.2%) white (U.S. census), 2,834 (8.1%) African American (U.S. census) . . . ”, the application instruction sequence “DIFF (9, SUM (10, 12))”, and the application result “6529”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Nishida by including a program synthesis (PS) model training unit that is taught by MA, to make the invention that generates a natural language processing (NLP) model that is a machine learning model that executes processing on a document written in a natural language; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving a natural language processing model using a neural network as well as enhancing the program output from the NLP model.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Relevant Prior Art Directed to State of Art
Izuka et al (U.S. 20100189366 A1), “Diagnosis Support Apparatus and Control Method Therefor”, teaches about a diagnosis support apparatus that supports the creation of findings in interpretation or image diagnosis. It also teaches about the apparatus acquires image feature information of a target area designated on an image to be interpreted, searches the storage unit for image feature information similar to the acquired image feature information, acquires a finding sentence stored in correspondence with the retrieved image feature information from the storage unit, and creates a finding sentence concerning interpretation of the designated target area by changing a description of the acquired finding sentence based on image feature information of the designated target area.
Lv et al (U.S 20210406619 A1), “ Method and Apparatus for Visual Question Answering, Computer Device and Medium”, teaches about a method for visual question answering, comprising: acquiring an input image and an input question; detecting visual information and position information of each of at least one text region in the input image; determining semantic information and attribute information of each of the at least one text region based on the visual information and the position information; determining a global feature of the input image based on the visual information, the position information, the semantic information, and the attribute information; determining a question feature based on the input question; and generating a predicted answer for the input image and the input question based on the global feature and the question feature.
Lubbers et al (U.S. 20180329892 A1), “Captioning a Region of an Image”, teaches about The method comprises providing a dataset of triplets. Each triplet includes a respective image, a respective region of the respective image, and a respective caption of the respective region. The method also comprises learning, with the dataset of triplets, a function that is configured to generate an output caption based on an input image and on an input region of the input image.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Duy A Tran whose telephone number is (571)272-4887. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL R MISTRY can be reached at (313)-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY TRAN/ Examiner, Art Unit 2674
/ONEAL R MISTRY/ Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights