DETAILED ACTION
Claims 1-20 are currently pending.
Response to Arguments
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-8, 11, 13-15, 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tewfik et al. US Publication 2011/0044521 (hereafter “Tewfik”) and Yu et al. US Publication 2019/0066294 (hereafter ”Yu”).
Referring to claims 1 and 17, Tewfik discloses a computer-implemented method for identifying a type of organ in a volumetric medical image, comprising:
a) receiving the volumetric medical image, the volumetric medical image comprising at least one organ or portion thereof (paragraph 183, At 610, the organ surface is sampled. Sampling can include generating data using an MRI modality, a CT scan, an ultrasound, a video camera or other system);
c) sampling, at the sampling locations, voxels from the volumetric medical image (paragraph 183, At 610, the organ surface is sampled. Sampling can include generating data using an MRI modality, a CT scan, an ultrasound, a video camera or other system), wherein at least one voxel is skipped between two sampled voxels (paragraph 78, The global sampling strategy uses single surface pixel uniformly distributed around the organ as shown in FIG. 3A); and
d) identifying the type of organ at the single point of interest by applying a trained classifier to the sampled voxels (paragraph 47, an example of the present subject matter identifies a specific, structured, sparse representation of the 3D organ surface that matches the very limited observed data and is suitable for a naturally shaped or a deformed organ).
While Tewfik discloses the volumetric medical image, Tewfik does not disclose expressly a sampling model that coincides with a single point of interest and that the trained classifier labels the single point of interest.
Yu discloses c) positioning a sampling model within the volumetric medical image such that a center of the sampling model coincides with the single point of interest (paragraph 100, The seed point determination sub-module 502 may determine one or more seed points of a rib. In some embodiments, a seed point may include a pixel/voxel defining an initial position for rib extraction), the sampling model specifying a predetermined set of sampling locations distributed about the center, and sampling, at the sampling locations of the sampling model centered on the single point of interest, voxels from the volumetric medical image (paragraph 101, In some embodiments, the rib pre-segmentation sub-module 504 may determine a portion of a plurality of pixels/voxels belonging to the rib(s) in the image data); and
d) constructing a descriptor comprising intensity values of the sampled voxels and identifying the type of organ at the single point of interest by applying a trained classifier to the descriptor, the trained classifier mapping the descriptor to a single label identifying the type of organ at the single point of interest (paragraph 112, In 507, the rib pre-segmented in 503, and/or the rib segmented in 505 may be labelled. In some embodiments, operation 507 may be performed by the rib labelling sub-module 508. In some embodiments, the rib may be labelled based on the seed point(s) determined in 501).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to adopt a sampling model and trained classifier that coincides with a single point of interest. The motivation for doing so would have been to improve the efficiency of classifying and labeling and improve visualization. Therefore, it would have been obvious to combine Yu with Tewfik to obtain the invention as specified in claims 1 and 17.
Referring to claim 5, Tewfik discloses wherein the sampled voxels are less than 1% of a total number of voxels in the volumetric medical image (paragraph 73, Each brain mesh can include 40962 points and spherical harmonics up to degree 80 can be used for approximation. Based on the training deformations, deformation subspaces can be identified using the ISI approach. The brain surface can be reconstructed by monitoring 29 sample positions).
Referring to claim 6, Tewfik discloses wherein the sampled voxels are less than 0.1 % of a total number of voxels in the volumetric medical image (paragraph 73, Each brain mesh can include 40962 points and spherical harmonics up to degree 80 can be used for approximation. Based on the training deformations, deformation subspaces can be identified using the ISI approach. The brain surface can be reconstructed by monitoring 29 sample positions).
Referring to claim 7, Tewfik discloses wherein the sampled voxels are less than 0.01 % of a total number of voxels in the volumetric medical image (paragraph 73, Each brain mesh can include 40962 points and spherical harmonics up to degree 80 can be used for approximation. Based on the training deformations, deformation subspaces can be identified using the ISI approach. The brain surface can be reconstructed by monitoring 29 sample positions).
Referring to claim 8, Tewfik discloses the trained classifier, but does not disclose expressly wherein the trained classifier is a neural network.
Yu discloses wherein the trained classifier is a neural network (paragraph 95, In some embodiments, the segmentation algorithms may include a neural network segmentation algorithm).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to use a neural network classifier. The motivation for doing so would have been to increase the efficiency and accuracy of classifying medical images. Therefore, it would have been obvious to combine Yu with Tewfik to obtain the invention as specified in claim 8.
Referring to claim 11, Yu discloses wherein the single point of interest is selected by a user (paragraph 109, As another example, a user may determine the seed point(s) through the I/O 230).
Referring to claim 13, Tewfik discloses wherein a user takes a measurement with respect to the volumetric medical image or a part thereof (paragraph 184, Sampling unit 720 can include an endoscope, a needlescope, a camera, or other instrument to collect sample measurements of a surface of an object (or organ)), and the identified type of organ is saved in a database along with the measurement (paragraph 9, Training data can be generated and used to construct dictionaries and identify subspaces).
Referring to claim 14, Tewfik discloses receiving a number of untrained classifiers for identifying organ specific abnormalities (paragraph 48, Training 110 can include constructing dictionaries and identifying subspaces using MRI, CT scans, ultrasound, or modeling);
selecting one or more of the untrained classifiers from the number of classifiers depending on the identified organ (paragraph 98, Information about the tumor shape and coarse consistency can be determined from the pre-operative scans to select the proper subspaces); and
training the one or more untrained classifiers using the volumetric medical image (paragraph 63, At 215, a subspace is learned from the training data).
Referring to claim 15, Tewfik discloses performing or repeating steps a) to d) for each of N-1 single points of interest within the volumetric medical image, wherein N is less than or equal to a total number of voxels of the volumetric medical image (paragraph 68, Scanning data for approximately 10% of an organ surface is sufficient for reconstruction).
Referring to claim 20, Tewfik discloses wherein step c) comprises sampling the voxels in a sparse or random manner (paragraph 42, Sparse surface and internal structure representations can be used in reconstruction methods based on limited view data from the surface of the organ).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Tewfik et al. US Publication 2011/0044521 and Yu et al. US Publication 2019/0066294 as applied to claims 1 and 17 above, and further in view of Yang et al. US Publication 2005/0285858 (hereafter “Yang.”).
Referring to claim 2, Tewfik discloses the sampling model and El-Zehiry discloses wherein a cube is centered on the single point of interest (paragraph 56, In act D, the clinician identifies an area of interest, for example, a chamber or a blood vessel of the heart. It may be sufficient for the clinician to position a cursor over the area of interest, (e.g., “inside” the cavity shown on the monitor), so that the coordinates of the cursor relative to the image may be noted).
Tewfik and Yu do not disclose expressly wherein the sampling model comprises multiple cubes nested within each other.
Yang discloses wherein the sampling model comprises multiple cubes nested within each other (paragraph 29, The top level node of an octree is referred to as the root node, which usually contains information about the entire dataset volume. Each intermediate layer node contains information about the nodes at the next lower level).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to provide multiple cubes nested within each other. The motivation for doing so would have been to increase the efficiency and speed of rendering volumetric images. Therefore, it would have been obvious to combine Yang with Tewfik and Yu to obtain the invention as specified in claim 2.
Claims 3 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Tewfik et al. US Publication 2011/0044521 and Yu et al. US Publication 2019/0066294 as applied to claims 1 and 17 above, and further in view of Brosch et al. US Publication 2020/0410691 (hereafter “Brosch”).
Referring to claims 3 and 18, Tewfik discloses wherein step c) comprises sampling the voxels with a sampling rate per unit length, area or volume (paragraph 183, At 610, the organ surface is sampled. Sampling can include generating data using an MRI modality, a CT scan, an ultrasound, a video camera or other system), but does not disclose expressly wherein the sampling decreases with a distance of a respective voxel from the single point of interest.
Brosch discloses wherein step c) comprises sampling the voxels with a sampling rate per unit length, area or volume which decreases with a distance of a respective voxel from the single point of interest (paragraph 25, the sampling rate can be reduced with increasing distance to the center of the respective surface element, wherein the resulting reduced overall sampling rate can lead to reduced computational efforts needed for segmenting the object in the image).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to decrease sampling with a distance from a single point of interest. The motivation for doing so would have been to increase efficiency by reducing computational efforts needed to segment the object in the image. Therefore, it would have been obvious to combine Brosch with Tewfik to obtain the invention as specified in claims 3 and 18.
Claims 4 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tewfik et al. US Publication 2011/0044521, Yu et al. US Publication 2019/0066294 and Brosch et al. US Publication 2020/0410691 as applied to claims 3 and 18 above, and further in view of Xu et al. Publication 2009/0087122 (hereafter “Xu”).
Referring to claims 4 and 19, Tewfik discloses sampling the voxels (paragraph 183, At 610, the organ surface is sampled. Sampling can include generating data using an MRI modality, a CT scan, an ultrasound, a video camera or other system).
Brosch discloses wherein the sampling rate decreases (paragraph 25, the sampling rate can be reduced with increasing distance to the center of the respective surface element, wherein the resulting reduced overall sampling rate can lead to reduced computational efforts needed for segmenting the object in the image).
Tewfik and Brosch do not disclose expressly wherein the sampling rate decreases at a non-linear rate.
Xu discloses wherein the sampling rate decreases at a non-linear rate (paragraph 42, The original video data may undergo a step of compression such as reducing the number of pixels in the frames by a non-linear spatial sub-sampling operation).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to decrease the sampling rate at a non-linear rate. The motivation for doing so would have been to reduce data processing to improve efficiency without sacrificing image analysis. Therefore, it would have been obvious to combine Xu with Tewfik and Brosch to obtain the invention as specified in claims 4 and 19.
Claims 9 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Tewfik et al. US Publication 2011/0044521 and Yu et al. US Publication 2019/0066294 as applied to claims 1 and 17 above, and further in view of El-Zehiry et al. US Publication 2021/0279884 (hereafter ”El-Zehiry”).
Referring to claim 9, Yu discloses the neural network, but does not disclose expressly wherein the neural network is a multilayer perceptron, a convolutional neural network, a Siamese network or a triplet network.
El-Zehiry discloses wherein the neural network is a multilayer perceptron, a convolutional neural network, a Siamese network or a triplet network (paragraph 10, In one example, the convolutional neural network is U-Net, but any other semantic segmentation architecture (such as a residual neural network, SegNet, DeepLab etc.) may be implemented).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to use a convolutional neural network classifier. The motivation for doing so would have been to increase the efficiency and accuracy of classifying medical images. Therefore, it would have been obvious to combine Tewfik and Yu with El-Zehiry to obtain the invention as specified in claim 9.
Referring to claim 12, Yu discloses selecting the single point of interest, but does not disclose expressly wherein the single point of interest is selected by pausing a cursor operated by a user.
El-Zehiry discloses wherein the single point of interest is selected by pausing a cursor operated by a user on the volumetric medical image or a part thereof displayed on a graphical user interface (paragraph 56, In act D, the clinician identifies an area of interest, for example, a chamber or a blood vessel of the heart. It may be sufficient for the clinician to position a cursor over the area of interest, (e.g., “inside” the cavity shown on the monitor), so that the coordinates of the cursor relative to the image may be noted).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to use a cursor to select the point of interest. The motivation for doing so would have been to simplify the manner in which a user analyses medical images. Therefore, it would have been obvious to combine Tewfik and Yu with El-Zehiry to obtain the invention as specified in claim 12.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tewfik et al. US Publication 2011/0044521 and El-Zehiry et al. US Publication 2021/0279884 as applied to claim 1 above, and further in view of He et al. US Publication 2021/0327563 (hereafter “He”).
Referring to claim 10, Tewfik discloses wherein the volumetric medical image or a part thereof comprising the single point of interest is displayed on a graphical user interface, wherein a semantic description of the identified type of organ is generated and displayed (paragraph 91, A living 3D reconstructed image will move on the monitor in real time as the organ itself or adipose tissue moves and may assist the surgeon in keeping track of a tumors location in relation to the organs surface and the location of blood vessels in adipose tissue, while manipulating and exposing the organ during an operation).
El-Zehiry discloses wherein the volumetric medical image or a part thereof comprising the single point of interest is displayed on a graphical user interface (paragraph 56, In act D, the clinician identifies an area of interest, for example, a chamber or a blood vessel of the heart. It may be sufficient for the clinician to position a cursor over the area of interest, (e.g., “inside” the cavity shown on the monitor), so that the coordinates of the cursor relative to the image may be noted).
Tewfik and El-Zehiry do not disclose expressly wherein a semantic description of the identified type of organ is generated and displayed at or adjacent to the single point of interest.
He discloses wherein the volumetric medical image or a part thereof comprising the single point of interest is displayed on a graphical user interface, wherein a semantic description of the identified type of organ is generated and displayed at or adjacent to the single point of interest (paragraph 120, Subsequent to such receipt, autonomously or via clinician activation, image viewer 51 proceeds to display the feature assessment data 31a in a textual format or a graphical format, and salient image generator 53 process salient image data 36a to generate salient image(s) 33 for display by image viewer 51).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to display a semantic description of the identified type of organ. The motivation for doing so would have been to inform the user of relevant information related to the medical image. Therefore, it would have been obvious to combine He with Tewfik and Yu to obtain the invention as specified in claim 10.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Tewfik et al. US Publication 2011/0044521, Yu et al. US Publication 2019/0066294 and Do et al. US Publication 2020/0285906 (hereafter “Do”).
Referring to claim 16, Tewfik discloses a computer-implemented method of training a classifier for identifying a type of organ in a volumetric medical image, comprising:
a) receiving the volumetric medical image, the volumetric medical image comprising at least one organ or portion thereof (paragraph 66, At 235, data is captured. The data capture can be accomplished using model data, ultrasound data, computerized tomography (CT), or magnetic resonance imaging (MRI) data. At 240), and receiving the type of the at least one organ (paragraph 64, An iterative subspace identification (ISI) method is used to learn the subspaces);
c) sampling voxels from the volumetric medical image (paragraph 183, At 610, the organ surface is sampled. Sampling can include generating data using an MRI modality, a CT scan, an ultrasound, a video camera or other system), wherein at least one voxel is skipped between two sampled voxels (paragraph 78, The global sampling strategy uses single surface pixel uniformly distributed around the organ as shown in FIG. 3A);
d) identifying the type of organ by applying an untrained classifier to the sampled voxels (paragraph 63, At 210, data from organ 205 is transformed using spherical harmonics. At 215, a subspace is learned from the training data);
While Tewfik discloses the volumetric medical image, Tewfik does not disclose expressly a sampling model that coincides with a single point of interest and that the trained classifier labels the single point of interest.
Yu discloses c) positioning a sampling model within the volumetric medical image such that a center of the sampling model coincides with the single point of interest (paragraph 100, The seed point determination sub-module 502 may determine one or more seed points of a rib. In some embodiments, a seed point may include a pixel/voxel defining an initial position for rib extraction), the sampling model specifying a predetermined set of sampling locations distributed about the center, and sampling, at the sampling locations of the sampling model centered on the single point of interest, voxels from the volumetric medical image (paragraph 101, In some embodiments, the rib pre-segmentation sub-module 504 may determine a portion of a plurality of pixels/voxels belonging to the rib(s) in the image data); and
d) constructing a descriptor comprising intensity values of the sampled voxels and identifying the type of organ at the single point of interest by applying a trained classifier to the descriptor, the trained classifier mapping the descriptor to a single label identifying the type of organ at the single point of interest (paragraph 112, In 507, the rib pre-segmented in 503, and/or the rib segmented in 505 may be labelled. In some embodiments, operation 507 may be performed by the rib labelling sub-module 508. In some embodiments, the rib may be labelled based on the seed point(s) determined in 501).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to adopt a sampling model and trained classifier that coincides with a single point of interest. The motivation for doing so would have been to improve the efficiency of classifying and labeling and improve visualization.
While Tewfik discloses applying an untrained classifier to the sampled voxels, Tewfik does not disclose expressly the training including e) comparing the identified type of organ with the received type of the at least one organ; and f) modifying the classifier depending on the comparison of step e) to obtain a trained classifier.
Do discloses e) comparing the identified type of organ with the received type of the at least one organ (paragraph 46, at step 350, the system may evaluate whether the network's ability to identify a feature in an image exceeds a defined threshold of speed, classification accuracy, reproducibility, efficacy, or other performance metric); and
f) modifying the classifier depending on the comparison of step e) to obtain a trained classifier (paragraph 47, If a desired level of performance is not achieved, then the labeled data may be refined at step 360).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to perform supervised training. The motivation for doing so have been to improve the results of the classifier by modifying incorrect classifications from the machine learning model. Therefore, it would have been obvious to combine Yu and Do with Tewfik to obtain the invention as specified in claim 16.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PETER K HUNTSINGER whose telephone number is (571)272-7435. The examiner can normally be reached Monday - Friday 8:30 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benny Q Tieu can be reached at 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER K HUNTSINGER/ Primary Examiner, Art Unit 2682
Prosecution Insights