Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 U.S.C. § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2 and 7 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Kadkhodamohammadi (“A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms”).
Regarding claim 1, Kadkhodamohammadi teaches a keypoint detection method for medical imaging (see abstract, human body-pose keypoint estimation of a subject in an operating room during live surgery), the method comprising acquiring a color image and a depth image of a subject, the color image being synchronized with the depth image in a temporal dimension (Fig. 1 and § 2.1, synchronized pairs of color and depth images recorded by an RGB-D camera system),
performing image fusion on the color image and the depth image, to generate a fused image (§ 2.1.1, the network input layer is changed to accept four-dimensional data, i.e., the three color channels and the depth channel joined into a single RGB-D input),
and performing keypoint detection on the fused image to generate keypoint distribution information of the subject (§ 2.1.1, a deep residual convolutional network body-part detector predicts confidence scores for all body parts at every image location, yielding the body-joint locations of the subject).
Regarding claim 2, see § 2.1.1 of Kadkhodamohammadi, the color image comprises N=3 color channels and the depth image comprises one depth channel, connected in the channel dimension to form the four-dimensional (N+1) fused RGB-D input.
Regarding claim 7, see § 2.1.1 of Kadkhodamohammadi, the keypoint detection is performed by means of a deep residual convolutional neural network (deep learning model).
Claim(s) 1, 4-6, 13-14 and 16-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Pai (20210251516).
Regarding claim 1, Pai teaches a keypoint detection method for medical imaging, the method comprising acquiring a color image and a depth image of a subject, the color image being synchronized with the depth image in a temporal dimension (Pai, pars. 26 and 33, acquiring RGBD images of the patient as a live stream using an RGBD sensor),
performing image fusion on the color image and the depth image to generate a fused image (par. 37, creating two or more channels including one channel for depth from the sensor and another channel for the optical/color image of the patient, combined as the channel data input),
performing keypoint detection on the fused image to generate keypoint distribution information of the subject (Pai, pars. 67-68, detecting a plurality of anatomical landmarks from the RGBD data using one or more machine-learned models and a trained regressor)
Regarding claim 4, see Pai, pars. 41-42, pre-processed data.
Regarding claim 5, see Pai, pars. 42 and 66, pre-processing of the depth image includes normalizing the data and aligning the data via reprojection to a common view.
Regarding claim 6, see Pai, par. 42, the color/surface data are normalized.
Regarding claim 13, see Pai, pars. 26 and 39, a color image sequence and a depth image sequence are acquired as a stream or sequence of RGBD camera captures, frames of surface data representing the patient at different times being acquired in a sequence, and the color image and the temporally-aligned depth image are taken from the respective sequences.
Regarding claim 14, see Pai, pars. 37 and 39, image fusion.
Regarding claim 16, Pai teaches a medical imaging method comprising see the rejection of claim 1 and also performing a scanning operation according to the determined keypoint distribution information (Pai, par. 80, patient is centered at the iso-center of the scanner based on the detected landmarks).
Regarding claim 17, Pai teaches the scanning operation includes positioning the subject according to the keypoint distribution information (par. 80, the bed moves the patient so that a center determined from the landmark-fit model is at the scanner iso-center).
Regarding claim 18, Pai teaches a medical imaging system comprising a controller (the image processor of the medical system, par. 30) configured to execute the keypoint detection method for medical imaging of claim 1 as mapped above, and a scanning assembly, performing a scanning operation according to keypoint distribution information determined by the controller (par. 30 and 80).
Regarding claim 19, see Pai par. 80.
Claim Rejections - 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Kadkhodamohammadi in view of Eitel (“Multimodal Deep Learning for Robust RGB-D Object Recognition”).
Regarding claim 3, Kadkhodamohammadi teaches the color image comprising N color channels and connecting the color image with a depth-derived image in the channel dimension to obtain the fused image (§ 2.1.1), but connects a single depth channel and does not expressly convert the depth image into a mapped color image comprising M color channels.
Eitel teaches converting a depth image into a mapped color image by means of color mapping, applying a jet colormap that transforms the single-channel depth image into a three-channel image, distributing the depth information over all three RGB channels (§ III-A).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to convert the depth image of Kadkhodamohammadi into a three-channel mapped color image as taught by Eitel and to connect it with the color image in the channel dimension (N+M channels). The reason is to spread the depth information across three channels so that standard pre-trained RGB convolutional networks can be leveraged for stronger feature extraction.
Claims 8-12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Pai in view of Kadkhodamohammadi.
Regarding claim 8, Pai teaches training the model on a preset training data set comprising many samples of frames of surface data with corresponding ground-truth landmark (keypoint distribution) information (par. 47), the input image being generated by acquiring a first color image and a first depth image of a subject and fusing them into the multi-channel input.
Kadkhodamohammadi teaches training a deep residual convolutional neural network on color-and-depth training images to detect body-part keypoints (§ 2.1.1).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to implement the trained model of Pai as the deep convolutional neural network of Kadkhodamohammadi, trained on the fused first color and first depth training images, in order to improve keypoint detection accuracy.
Regarding claim 9, see Pai, pars. 42 and 66.
Regarding claim 10, see Pai pars. 41-42.
Regarding claim 11, Pai teaches performing keypoint (landmark) detection and generating keypoint distribution information from combined color-and-depth data by means of a machine-learned model (pars. 46 and 67).
Kadkhodamohammadi teaches a deep convolutional neural network that jointly receives and processes the color and depth channels as its input and thereby learns fused color/depth features from which it outputs the keypoint confidence score maps (§ 2.1.1).
It would have been obvious prior to the effective filing date of the invention to one of ordinary skill in the art to have the deep learning model jointly process the color and depth channels to perform the fusion and keypoint detection as taught by Kadkhodamohammadi, in order to learn complementary color and depth features end-to-end and improve keypoint detection accuracy.
Regarding claim 12, see Pai, par. 47
Regarding claim 15, Pai teaches acquiring the fused image sequence and repeatedly performing landmark detection on the frames over the stream (par. 67, landmarks regressed at each time instance), generating the keypoint distribution information of the subject over time.
Kadkhodamohammadi teaches performing keypoint detection to produce per-image body-part confidence score maps (i.e., keypoint images) (§ 2.1.1).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure: Zimmermann (“3D Human Pose Estimation in RGBD Images for Robotic Task Learning”) teaches 3D human keypoint estimation from RGB-D images by fusing color-derived 2D keypoint score maps with a depth-derived voxel occupancy grid in a 3D convolutional network.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HADI AKHAVANNIK whose telephone number is (571)272-8622. The examiner can normally be reached 9 AM - 5 PM Monday to Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HADI AKHAVANNIK/Primary Examiner, Art Unit 2676
Prosecution Insights