DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) filed on 1/18/2024 was considered and placed on the file of record by the examiner.
Claim Objections
Claim 9 objected to because of the following informalities: missing a period. Appropriate correction is required.
Drawings
The drawings are objected to under 37 CFR 1.83(a) because they fail to show details as described in the specification. Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing. MPEP § 608.02(d). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 13 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 13 recites the limitation " the slice order” not defined or distinguished from a slice score that is the position of a slice. There is insufficient antecedent basis for this limitation in the claim. Non-application of art to Claim 13 is an indication of the severity of the 35 USC 112 rejection and not an indication of allowability.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-7, 11, 12, 14, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Han (US 2019/0030371) in view of Sakaida (US 2010/0002917).
Regarding claim 1, Han teaches a computer-implemented method comprising:
receiving at least one image, the image representing a slice, the slice being oriented perpendicular to an axis of a volume of an object (see para. 0021, 0054, 0064, Han discusses acquiring medical images that are slices along the rotational axis of an object);
inputting the at least one image into a first model (see figure 9, figure 10, para. 0021-0022, Han discusses inputting image slices into a first portion of a CNN);
receiving from the second model an object part information, the object part information indicating to which part/parts of the object the slice belongs to (see figure 9, figure 10, para. 0021-0022, 0093-0094, 0098, Han discusses a second model extracts label data for the region of interest from a first model); and
outputting and or storing the object part information and/or information related thereto (see figure 5, figure 9, figure 10, para. 0093-0094, 0098, Han discusses generating segmentation data of a region of interest of an organ structure).
Sakaida teaches receiving, from the first model, for each image inputted into the first model, a slice score, the slice score being representative of the position of the slice within the object along the axis, inputting the slice score into a second model (see para. 0068, 0072, Sakaida discusses receiving a score table for image slices, and an order of slices based on the image position information).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 1. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Han in this manner in order to improve object segmentation on medical slices taking into account the position of the slices to narrow and optimize the segmentation process of a model by focusing on a region of interest instead of entire human body. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Han, while the teaching of Sakaida continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of calculating slice orders to optimize model segmentation of medical slice images. The Han and Sakaida systems perform medical slice image analysis, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 2, Han teaches wherein the first model is or comprises a machine learning model which was trained on training data to determine slice scores on the basis of images, wherein the training data comprise, for each object of a multitude of objects (see para. 0067-0068, Han discusses training the CNN to stack slices in sequence);
a set of reference images, each reference image representing a slice along an axis of a volume of the object, each slice being oriented perpendicular to the axis, and a slice order, the slice order indicating the order in which the slices follow one each other along the axis of the volume of the object (see para. 0067-0068, Han discusses stack of adjacent 2D images along the axis orthogonal to the anatomical plane of the 2D images).
The same motivation of claim 1 is applied to claim 2. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 2. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Regarding claim 3, Han teaches wherein the second model is configured to determine, based on the slice score, an object part information, wherein the object part information indicates to which part/parts of the object the slice belongs to (see para. 0067-0068, Han discusses providing information of the shape, size, or type of the anatomical structure in another adjacent image along the same plane).
The same motivation of claim 1 is applied to claim 3. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 3. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Regarding claim 4, Sakaida teaches further comprising: receiving a set of images, wherein the set of images comprises a plurality of images, each image representing a slice, each slice being oriented perpendicular to an axis of a volume of an object, the object being divided into different parts (see para. 0002, Sakaida discusses stack of adjacent 2D axial images perpendicular to the body axis);
inputting each image into the first model, receiving, from the first model, for each image inputted into the first model, a slice score, the slice score being representative of the position of the slice within the object along the axis (see figure 2, figure 8, figure 9, para. 0094-0095, 0114, Sakaida discusses part recognition unit that determines three-dimensional coordinates of the slice image),
inputting one or more slice scores into the second model, receiving from the second model, for each slice score inputted into the second model, an object part information, the object part information indicating to which part/parts of the object the slice belongs to (see para. 0100, 0102, 0119-0121, Sakaida discusses part segmentation unit that detects a boundary surface where the part changes in the z-axis direction), and
combining the object part information with the respective image and storing the respective image together with the object part information in a data storage (see figure 1, figure 2, para. 0108, Sakaida discusses a storage unit for storing organ slice image data).
The same motivation of claim 1 is applied to claim 4. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 4. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Regarding claim 5, Sakaida teaches further comprising:
receiving a 3D representation of the volume of the object (see para. 0093, Sakaida discusses three-dimensional information of the object represented by the set of plural slice images);
generating a set of 2D images from the 3D representation, each 2D image representing a slice, each slice being oriented perpendicular to a defined axis of the volume of the object (see figure 5, para. 0002, 0074, 0093, Sakaida discusses set of plural slice images that are axial image that shows a surface perpendicular to the body axis);
inputting each 2D image into the first model (see figure 1, figure 2, para. 0010, Sakaida discusses inputting slice images into part information unit);
receiving, from the first model, for each 2D image inputted into the first model, a slice score, the slice score being representative of the position of the slice within the object along the axis (see figure 1, figure 2, para. 0010, Sakaida discusses inputting slice images along the body axis into part information unit; see para. 0033-0034, 0068-0072, Sakaida discusses receiving a score table for image slices, and a part information unit with the order of slices based on the image position information order of slices, based on the image position information);
inputting one or more slice scores into the second model (see figure 1, figure 2, para. 0033-0034, 0068-0072, Sakaida discusses inputting slice images into part information unit);
receiving, from the second model, for each slice score inputted into the second model, an object part information, the object part information indicating to which part/parts of the object the slice belongs to (see figure 1, figure 2, para. 0113, 0122, 0125, Sakaida discusses inputting slice scores and positional coordinate data into segmentation unit); and
combining the object part information with the respective 2D image and storing the respective 2D image together with the object part information in a data storage (see figure 1, figure 2, para. 0044, 0108, Sakaida discusses a storage unit for storing slice image data for organs, features, and positional information).
The same motivation of claim 1 is applied to claim 5. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 5. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Regarding claim 6, Han teaches further comprising: receiving a first set of 2D images, the first set of 2D images representing a stack of slices of a volume of an object, wherein the slices are not oriented perpendicular to a defined axis of the volume of the object (see para. 0044, Han discusses receiving 2D slices that may include a sagittal orientation, a coronal orientation, or an axial orientation);
generating a 3D representation of the volume from the first set of 2D images (see figure 9, para. 0091, Han discusses generating a 3D representation of 2D slices).
Sakaida teaches generating a set of 2D images from the 3D representation, each 2D image of the set of 2D images representing a slice, each slice being oriented perpendicular to the defined axis of the volume of the object (see figure 5, para. 0002, 0074, 0093, Sakaida discusses set of plural slice images that are axial image that shows a surface perpendicular to the body axis);
inputting each 2D image into the first model (see figure 1, figure 2);
receiving, from the first model, for each 2D image inputted into the first model, a slice score, the slice score being representative of the position of the slice within the object along the axis; inputting one or more slice scores into the second model (see figure 1, figure 2, para. 0010, Sakaida discusses inputting slice images along the body axis into part information unit; see para. 0033-0034, 0068-0072, Sakaida discusses receiving a score table for image slices, and a part information unit with the order of slices based on the image position information order of slices, based on the image position information);
receiving, from the second model, for each slice score inputted into the second model, an object part information, the object part information indicating to which part/parts of the object the slice belongs to (see figure 1, figure 2, para. 0113, 0122, 0125, Sakaida discusses inputting slice scores and positional coordinate data into segmentation unit); and
combining the object part information with the respective 2D image and storing the respective 2D image together with the object part information in a data storage (see figure 1, figure 2, para. 0044, 0108, Sakaida discusses a storage unit for storing slice image data for organs, features, and positional information).
The same motivation of claim 1 is applied to claim 6. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 6. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Regarding claim 7, Han teaches comprising the steps: receiving a plurality of images, each image representing a slice, each slice being oriented perpendicular to an axis of a volume of an object, the object being divided into different parts (see para. 0092, Han discusses 3D image slice-by-slice along the axis of the body);
inputting each image into the first model (see para. 0092, Han discusses applying images into a first model);
receiving, from the first model, for each image inputted into the first model, a slice score, the slice score being representative of the position of the slice within the object along the axis (see para. 0092, Han discusses the first model determines the positional data to limit the segmentation of a smaller region of interest);
inputting two limiting slice scores into the second model, the limiting slice scores limiting the volume along the axis (see para. 0092, Han discusses inputting the position data of the limited smaller region of interest into the second model).
Sakaida teaches receiving, from the second model, an object part information, the object part information indicating to which part/parts of the object the volume is covering (see figure 1, figure 2, para. 0113, 0122, 0125, Sakaida discusses inputting slice scores and positional coordinate data into segmentation unit); and
combining the object part information with the set of images and storing the respective set of images together with the object part information in a data storage (see figure 1, figure 2, para. 0044, 0108, Sakaida discusses a storage unit for storing slice image data for organs, features, and positional information).
The same motivation of claim 1 is applied to claim 7. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 7. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Regarding claim 11, Han teaches wherein the object is a human being or an animal or a plant or a part thereof, preferably a human being (see figure 1A, para. 0011, Han discusses segmented three-dimensional (3D) computed tomography (CT) image from a pelvic region of a prostate cancer patient).
The same motivation of claim 1 is applied to claim 11. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 11. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Regarding claim 12, Han teaches wherein each image is a medical image (see figure 1A, figure 3, para. 0011, 0015, Han discusses segmented three-dimensional (3D) computed tomography (CT) image from a pelvic region of a prostate cancer patient).
The same motivation of claim 1 is applied to claim 12. Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han with Sakaida to derive at the invention of claim 12. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
Claim 14 is rejected as applied to claim 1 as pertaining to a corresponding computer system.
Claim 15 is rejected as applied to claim 1 as pertaining to a corresponding non-transitory computer readable medium.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Han (US 2019/0030371) in view of Sakaida (US 2010/0002917) in view of Yan et al. “Unsupervised Body Part Regression via Spatially Self-ordering Convolutional Neural Networks.”
Regarding claim 8, Han and Sakaida do not expressly disclose wherein the first model was trained in a training process comprising: receiving a training data set, the training data set comprising, for each object of a multitude of objects: a set of reference images, each reference image representing a slice along an axis of a volume of the object, each slice being oriented perpendicular to the axis of the volume of the object, and a slice order, the slice order indicating the order in which the slices follow each other along the axis of the volume of the object; inputting the reference images into the first model; receiving, from the first model, a slice score for each reference medical image inputted into the first model, the slice score representing the position of the slice along the axis; computing a loss value on the basis of the slice scores and the slice order using a loss function L, the loss function L comprising at least an order loss term L.sub.order, a distance loss term L.sub.dist, and a slice-gap loss term L.sub.slice-gap, wherein: the order loss term L.sub.order penalizes first model parameters which result in a sequence of slice scores according to their magnitude which does not correspond to the score order, the distance loss term L.sub.dist penalizes first model parameters which result in slice scores for which the differences of two pairs of equidistant slices are not equal, and the slice-gap loss term L.sub.slice-gap penalizes first model parameters which result in slice scores for which the difference between two slice scores is not proportional to the physical distance between the two slices; and modifying first model parameters in a way that reduces the loss value to a defined minimum.
However, Yan teaches wherein the first model was trained in a training process comprising: receiving a training data set, the training data set comprising, for each object of a multitude of objects (see abstract, Yan discusses a training dataset of images of CT volumes);
a set of reference images, each reference image representing a slice along an axis of a volume of the object, each slice being oriented perpendicular to the axis of the volume of the object, and a slice order, the slice order indicating the order in which the slices follow each other along the axis of the volume of the object (see figure 1, section 2, Yan discusses slice order along the axis of the volume);
inputting the reference images into the first model (see figure 1, section 2, Yan discusses a training dataset of images input into a CNN model);
receiving, from the first model, a slice score for each reference medical image inputted into the first model, the slice score representing the position of the slice along the axis (see figure 1, section 2, Yan discusses slice score representing the position of the slice along the axis);
computing a loss value on the basis of the slice scores and the slice order using a loss function L, the loss function L comprising at least an order loss term L.sub.order, a distance loss term L.sub.dist, and a slice-gap loss term L.sub.slice-gap (see figure 1, section 2, Yan discusses order loss, distance loss, and L1 loss), wherein:
the order loss term L.sub.order penalizes first model parameters which result in a sequence of slice scores according to their magnitude which does not correspond to the score order, the distance loss term L.sub.dist penalizes first model parameters which result in slice scores for which the differences of two pairs of equidistant slices are not equal, and the slice-gap loss term L.sub.slice-gap penalizes first model parameters which result in slice scores for which the difference between two slice scores is not proportional to the physical distance between the two slices (see figure 1, section 2, Yan discusses order loss, distance loss, and L1 loss are used to perform backpropagation that optimizes the network, the slice scores should be equidistant); and
modifying first model parameters in a way that reduces the loss value to a defined minimum (see figure 1, section 2, Yan discusses performing backpropagation to optimizes parameters of the network).
Motivation to combine may be gleaned from the prior art considered. It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the invention of Han and Sakaida with Yan to derive at the invention of claim 8. The result would have been expected, routine, and predictable in order to perform object segmentation on medical slices.
The determination of obviousness is predicated upon the following: One skilled in the art would have been motivated to modify Han and Sakaida in this manner in order to improve object segmentation on medical slices taking into account the position of the slices to narrow and performing a neural network loss optimization to optimize the segmentation process. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in this manner explained using known engineering design, interface and/or programming techniques, without changing a fundamental operating principle of Han and Sakaida, while the teaching of Yan continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of calculating a slice order and performing a loss function minimization to optimize model segmentation of medical slice images. The Han, Sakaida, and Yan systems perform medical slice image analysis, therefore one of ordinary skill in the art would have reasonable expectation of success in the combination. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Allowable Subject Matter
Claims 9, 10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: No prior art was found to claim “9. The method of claim 2, the first model was trained in a training process comprising: generating, for a plurality of reference images, a plurality of low sampling images; using the low sampling images as additional training data; computing a loss value on the basis of the slice scores and the slice order using a loss function L, the loss function L comprising at least a down-sampling loss term L.sub.down-sampling, the down-sampling loss term L.sub.down-sampling rewards first model parameters which result in equal slice scores for low-sampling images and reference images the low-sampling images were generated from; and modifying first model parameters in a way that reduces the loss value to a defined minimum”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ferl et al. (US 2023/0005140) discusses two-dimensional slice images correspond to consecutive and/or adjacent slices of the biological structure depicted in the training image. The trained machine-learning model performs classification on input images.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KENNY A CESE whose telephone number is (571) 270-1896. The examiner can normally be reached on Monday – Friday, 9am – 4pm.
If attempts to reach the primary examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached on (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kenny A Cese/
Primary Examiner, Art Unit 2663