DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 11, 15 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Gheorghita et al (US 20220093270 A1).
Regarding claim 1, Gheorghita discloses a data processing apparatus comprising processing circuitry (Gheorghita [0112]) configured to:
generate first artificial data (Gheorghita fig. 5; [0049], “generate datasets … generate new (generate artificial) (ED, ES) pairs where ED (an equivalent of a first artificial data)”);
generate second artificial data (Gheorghita fig. 5; [0049], “generate datasets … generate new (generate artificial) (ED, ES) pairs where … and ES (an equivalent of a second artificial data) is smaller.”);
generate mixed data by mixing the first artificial data and the second artificial data (Gheorghita [0049], “synthesized (ED, ES) pairs (mixed first artificial data and second artificial data pairs are generated) are used to create new synthetic patients with larger EF.”); and
generate mixed region information related to a region where the first artificial data and the second artificial data are mixed (Gheorghita fig. 6; [0052], “FIG. 6 shows some example synthetically generated masks … synthetic images are generated (generate mixed region information related to a region where the first artificial data and the second artificial data are mixed)”).
Gheorghita does not expressly disclose artificial data.
However, Gheorghita discloses generating synthetic data as synthetic masks (Gheorghita [0046], “synthetic data is generated”; [0047], “The masks are synthesized for different EF to then create the scan data (e.g., cardiac MRI).”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to realize the use of synthetic data as artificial data to further generate composite synthetic images. This would have been done to generate desired output data in an efficient and cost effective manner. See, for example, Gheorghita [0045], “Since the generation of synthetic data can be automated, the cost of generating a large database is reduced.”)
Regarding claim 2, Gheorghita discloses the data processing apparatus according to claim 1, wherein the mixed data and the mixed region information are training data for training a machine learning model (Gheorghita [0052], “The generated synthetic images are generated to be used as inputs to the model to be trained to estimate the EF.”).
Regarding claim 3, Gheorghita discloses the data processing apparatus according to claim 1, wherein:
the first artificial data are first artificial images (Gheorghita fig. 5; [0049], “generate datasets … generate new (ED, ES) pairs where ED (an equivalent of a first artificial data are first artificial images)”);
the second artificial data are second artificial images (Gheorghita fig. 5; [0049], “generate datasets … generate new (ED, ES) pairs where … and ES (an equivalent of second artificial data are second artificial images) is smaller.”); and
the mixed data are mixed artificial images in which the first artificial images and the second artificial images are respectively mixed (Gheorghita fig. 6; [0052], “FIG. 6 shows some example synthetically generated masks … synthetic images are generated (mixed data are mixed artificial images in which the first artificial images and the second artificial images are respectively mixed)”).
Regarding claim 4, Gheorghita discloses the data processing apparatus according to claim 1, wherein:
the processing circuitry is configured to generate the first artificial data and the second artificial data, by using respective generative models based on machine learning (Gheorghita [0063], “Any generative architecture may be used for unsupervised learning to predict segmentation, EF, and/or other anatomical or functional characteristics.”); and
a generative model used for generating the first artificial data and another generative model used for generating the second artificial data are different from each other in at least one of (a) type of the generative model, (b) a generation parameter used inside the respective generative models, and (c) a pseudorandom number sequence inputted to the respective generative models (Gheorghita [0048], “the interpolated mask is generated as: Interpolated mask=(α*SDT.sub.1)+((1−α)*SDT.sub.2) where SDT.sub.1 and SDT.sub.2 are signed distance transform masks of ED and ES of the actual patient, and a is a parameter provided as input, which may take values between 0 and 1 (generation parameter used inside the respective generative models). Thus, new pairs of ED and ES are formed: (ED, interpolated mask) and (interpolated mask, ES). FIG. 4 shows an example of synthesized new pairs where the actual mask is used for part of each pair. (separate models for each mask are used and the models differ based on different parameters)”).
Regarding claim 5, Gheorghita discloses the data processing apparatus according to claim 1, wherein:
the processing circuitry is configured to generate a first artificial image and a second artificial image, by using respective generative models based on machine learning (Gheorghita [0063], “Any generative architecture may be used for unsupervised learning to predict segmentation, EF, and/or other anatomical or functional characteristics.”); and
a generative model used for generating the first artificial image and another generative model used for generating the second artificial image are different from each other in at least one of (a) type of the generative model, (b) a generation parameter used inside the respective generative models, and (c) a pseudorandom number sequence inputted to the respective generative models (Gheorghita [0048], “the interpolated mask is generated as: Interpolated mask=(α*SDT.sub.1)+((1−α)*SDT.sub.2) where SDT.sub.1 and SDT.sub.2 are signed distance transform masks of ED and ES of the actual patient, and a is a parameter provided as input, which may take values between 0 and 1 (generation parameter used inside the respective generative models). Thus, new pairs of ED and ES are formed: (ED, interpolated mask) and (interpolated mask, ES). FIG. 4 shows an example of synthesized new pairs where the actual mask is used for part of each pair. (separate models for each mask are used and the models differ based on different parameters)”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Gheorghita with Weiss to utilize simulated foreground and background images for generating simulated composite images. This would have been done to generate images for a variety useful scenarios.
Regarding claim 11, Gheorghita discloses the data processing apparatus according to claim 1, wherein the processing circuitry is configured to further generate a first trained model by machine learning in which a plurality of mixed data generated in advance and a plurality of sets of the mixed region information generated in advance are used as training data (Gheorghita [0009], “he initial model (e.g., second model) may have been trained with many more samples, such as at least 1,000 samples. There may be very few (e.g., less than 100) samples from actual patients for the few shot learning. The number of samples may be increased by generating synthetic examples. Similarly, the number of samples for training the initial model may be increased by generating synthetic examples. For example, the at least 1,000 samples include a first set of samples from people and a second set of samples including the synthetic examples.”).
Regarding claim 15, Gheorghita discloses the data processing apparatus according to claim 1, wherein the processing circuitry is configured to generate the first artificial data and the second artificial data by using a machine learning model capable of generating artificial images, the machine learning model including at least one of a GAN (Generative Adversarial Network), a VAE (Variable Autoencoder), a Diffusion Model, and an IFS (Iterated Function System) (Gheorghita [0047], “generative adversarial network (GAN)”).
Claim 16 recites a medical image processing apparatus which corresponds to the function performed by the data processing apparatus of claim 1. As such, the mapping and rejection of claim 1 above is considered applicable to the medical image processing apparatus of claim 16.
Additionally, Gheorghita discloses a medical image processing apparatus (Gheorghita [0031], “The method is implemented by a machine (e.g., computer, processor… the method for other medical”).
Claim 17 recites a method which corresponds to the function performed by the data processing apparatus of claim 1. As such, the mapping and rejection of claim 1 above is considered applicable to the method of claim 17.
Claims 6-9 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Gheorghita in view of Weiss et al (US 20070165966 A1).
Regarding claim 6, Gheorghita discloses the data processing apparatus according to claim 3, but does not disclose wherein:
the first artificial image is a background artificial image that simulates a background of a segmentation target image;
the second artificial image is an artificial image that simulates an image of a segmentation target region in the segmentation target image; and
the processing circuitry is configured to generate, as the mixed artificial image, an image in which the background artificial image and a simulated artificial image are mixed and generate, as the mixed region information, a specific-region artificial image corresponding to the segmentation target region.
However, Weiss discloses
the first artificial image is a background artificial image that simulates a background of a segmentation target image (Weiss fig. 12c; [0127], “the other matte (interpreted as reading on first artificial image is a background artificial image that simulates a background of a segmentation target image) is a part of a circle, mostly opaque with a feathered boundary.”);
the second artificial image is an artificial image that simulates an image of a segmentation target region in the segmentation target image (Weiss fig. 12c; [0127], “the first matte (interpreted as reading on second artificial image is an artificial image that simulates an image of a segmentation target region in the segmentation target image) is the computer simulated smoke”; and
the processing circuitry is configured to generate, as the mixed artificial image, an image in which the background artificial image and a simulated artificial image are mixed and generate, as the mixed region information, a specific-region artificial image corresponding to the segmentation target region (Weiss [0127], “we obtained 4000 composite images, two of which are shown in FIG. 12(a) (mixed region information, a specific-region artificial image corresponding to the segmentation target region).)”).
Regarding claim 7, Gheorghita in view of Weiss discloses the data processing apparatus according to claim 6, wherein the second artificial image is a specific-region artificial image generated in such a manner that the segmentation target region and other regions can be distinguished by transparency information (Weiss [0127], “alpha matte … the computer simulated smoke most of which is partially transparent (regions can be distinguished by transparency information, alpha transparency); the other matte is a part of a circle, mostly opaque with a feathered boundary.”).
Regarding claim 8, Gheorghita in view of Weiss discloses the data processing apparatus according to claim 6, wherein the processing circuitry is configured to generate the mixed artificial image by mixing the first artificial image with the second artificial image where the region information is added, using the first artificial image, the second artificial image, and region information corresponding to a segmentation target region defined for the second artificial image (Weiss [0127], “alpha matte … the computer simulated smoke most of which is partially transparent (interpreted as reading on adding region information according the alpha transparency); the other matte is a part of a circle, mostly opaque with a feathered boundary.”).
Regarding claim 9, Gheorghita in view of Weiss discloses the data processing apparatus according to claim 6, wherein the background artificial image and the artificial image that simulates an image of the segmentation target region are generated to be different in statistical property (Gheorghita [0047], “where datasets for EF are synthesized based on a statistical model,”).
Regarding claim 12, Gheorghita in view of Weiss discloses the data processing apparatus according to claim 6, wherein the processing circuitry is configured to further generate a first trained model by machine learning in which a plurality of background artificial images generated in advance and a plurality of sets of specific region images generated in advance are used as training data (Gheorghita [0040], “The medical data of interest may the MRI scans (e.g., scan data), images from the MRI scanning (e.g., scan data), other imaging data”; [0041], “MRI scans may be available for hundreds or thousands of past patients”; [0087], “one or more tumor volumes (e.g., gross tumor volume) or regions including the tumor with or without non-tumor tissue are segmented.”).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Gheorghita in view of Cutler et al (US 20230117603 A1).
Regarding claim 10, Gheorghita discloses the data processing apparatus according to claim 3, further but does not disclose comprising a first trained model that has been trained to segment a region corresponding to the second artificial image in the mixed artificial image when the mixed artificial image is inputted.
However, Cutler discloses
a first trained model that has been trained to segment a region corresponding to the second artificial image in the mixed artificial image when the mixed artificial image is inputted (Cutler [0092], “foreground objects (users) were segmented from an artificial background in the image by an image segmentation model”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Gheorghita with Cutler segment artificial background images from an artificial image. This would have been done to separate image portions that can be reused to generate different images by utilizing the segmented image portions.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Gheorghita in view of Gordon et al (US 20220405915 A1).
Regarding claim 13, Gheorghita discloses the data processing apparatus according to claim 11, but does not disclose wherein the processing circuitry is configured to further generate a second trained model by applying transfer learning to the first trained model, the transfer learning being learning in which a plurality of real images and annotation information added to each of the plurality of real images are used as training data.
However, Gordon
the processing circuitry is configured to further generate a second trained model by applying transfer learning to the first trained model, the transfer learning being learning in which a plurality of real images and annotation information added to each of the plurality of real images are used as training data (Gordon [0104], “using a transfer learning approach … A baseline ML model is first trained on a baseline training dataset that includes the non-contrast medical images, each labelled with the ground truth label of the respective calcification parameter which was determined for that respective non-contrast medical image, as described herein. The ML model is then trained by further training the baseline ML model, using the transfer learning approach, on the training dataset that includes the contrast enhanced medical images, each labelled with ground truth label of the respective calcification parameter which were determined for the corresponding non-contrast medical image.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Gheorghita with Gordon to utilize transfer learning for training. This would have been done to improve training models in an iterative manner.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Gheorghita in view of Weiss and further view of Gordon.
Regarding claim 14, Gheorghita in view of Weiss discloses the data processing apparatus according to claim 12, but does not disclose wherein the processing circuitry is configured to further generate a second trained model by applying transfer learning to the first trained model, the transfer learning being learning in which a plurality of medical images and segmentation information added to each of the plurality of medical images are used as training data.
However, Gordon discloses
the processing circuitry is configured to further generate a second trained model by applying transfer learning to the first trained model, the transfer learning being learning in which a plurality of medical images and segmentation information added to each of the plurality of medical images are used as training data (Gordon [0104], “using a transfer learning approach … A baseline ML model is first trained on a baseline training dataset that includes the non-contrast medical images, each labelled with the ground truth label of the respective calcification parameter which was determined for that respective non-contrast medical image, as described herein. The ML model is then trained by further training the baseline ML model, using the transfer learning approach, on the training dataset that includes the contrast enhanced medical images, each labelled with ground truth label of the respective calcification parameter which were determined for the corresponding non-contrast medical image.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Gheorghita with Gordon to utilize transfer learning for training. This would have been done to improve training models in an iterative manner.
Conclusion
See the notice of references cited (PTO-892) for prior art made of record, including art that is not relied upon but considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JITESH PATEL whose telephone number is (571)270-3313. The examiner can normally be reached 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said A. Broome can be reached at (571) 272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JITESH PATEL/Primary Examiner, Art Unit 2612
Prosecution Insights