Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA
Information Disclosure Statement
The information disclosure statements (IDS) were submitted on 03/21/2024, 01/09/2025, 02/19/2025, 02/04/2026, 02/04/2026 and 03/12/2026. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant first argues the 112b rejections of claims 1-5, 7-9, 11-16, 18-22 and 24. Examiner revises the amendments applied and agrees with the rescindment of the 112b rejections. Claim 22 is now also allowable and objected to in view of the prior art and updated search. Applicant then argues the 102/103 rejection by mentioning that the claim limitation “registration between the two-dimensional X-ray image and the three-dimensional CT image” is not anticipated by reference Makoto. Examiner disagrees and shows reasoning. In the claim rejections of the independent claims the language utilized is the following: (Examiner interprets the following as all being part of the registration process. See all of page 3 and page 3 paragraph 7 “ The projection processing unit 3 receives the three-dimensional image data IM1 output from the three-dimensional image input unit 1 and the two-dimensional image data IM2 output from the two-dimensional image input unit 2, and uses IM1 as two-dimensional image data IM3 equivalent to IM2. (Step S400).” Therefore IM3 represents the CT image. See all of page 4, and page 4 paragraphs 3-4, the registration is performed “The registration unit 4 performs registration between the two-dimensional image data IM2 output from the two-dimensional image input unit 2 and the two-dimensional image data IM3 output from the projection processing unit 3, and results in an image after registration. IM2 ′ is output to the composition processing unit 5 and the accuracy evaluation unit 8 (step S500).” See also page 4 paragraph 11. See also page 5 paragraph 1. Makoto), reference Makoto teaches that IM1 is the 3D image data and converts the 3D image data into two-dimensional data IM3 and then performs a registration between the 2D x-ray image and the 3D image data represented as 2D IM3. This is within a BRI (broadest reasonable interpretation) in light of the specification of the claim language since it is creating a projected image of the 3D data to then perform the registration as it is commonly used within the art for 2D/3D registration, which is the same as already stipulated in the Applicant’s specification in paragraph 59 “[0059] The registration processing unit 30 receives the pair of the simple X-ray image 1 and the CT image 2 acquired by the information acquisition unit 28 as an input and performs registration between the simple X-ray image 1 and the CT image 2. A registration process of the registration processing unit 30 can be performed by, for example, projecting the three-dimensional CT image 2 to be converted into a two-dimensional DRR image and adjusting geometric conditions in the projection, the rotation of the CT image 2, and the like such that the converted DDR image is matched with the simple X-ray image 1.”. Therefore, independent claims 1 and 16 stay rejected because they are anticipated by Makoto.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 12, 16 and 17, are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Makoto et. al., hereafter Makoto (JP Publication no. 2007-159643 A).
As per claim 1, “A disease label creation device comprising: a first processor, wherein the first processor is configured to execute: (See page 5 paragraphs 4-12 “First, the computer 1000 will be described. A CPU 1010 controls the entire computer 1000 using programs and data stored in the RAM 1020 and the ROM 1030, and executes each process performed by the image processing apparatus” Makoto)
“an information acquisition process of acquiring a two-dimensional X-ray image, a three-dimensional CT image paired with the two-dimensional X-ray image, and a three-dimensional first disease label extracted from the three-dimensional CT image;” (See page 4 paragraphs 2-4 and 11. “By the above processing, the projection processing imitating the photographing means of the two-dimensional image data IM2 can be added to the three-dimensional image data IM1 and converted into the projection image data IM3 that is two-dimensional image data. In the present embodiment, the three-dimensional image IM1 is chest CT image data, and the two-dimensional image data IM2 is chest image data of the same subject, and the chest CT image data is converted into chest image data by projection processing.” IM2 is the two-dimensional x-ray image paired with IM1, which is the 3D CT image. IM2 is an x-ray image as seen in page 3 paragraphs 1, 5, 6, 9, 10 and in the abstract. The 3d first disease label (represents area) is structure data 0 seen in paragraph 11 (penultimate paragraph), “ The composite image IM4 is obtained by converting the structure data O having position information into a position on the two-dimensional image data IM3, and simply superimposing it on IM2 ′ aligned with IM3 to output the three-dimensional image. The notable structure O extracted in advance by IM1 is mapped and displayed on the two-dimensional image IM2.” See also page 3 paragraph 7. See also page 3 paragraph 3 “Next, the information extraction unit 7 extracts a specific structure O displayed in the 3D image data output from the 3D image input unit 1 (step S300).” Makoto)
“a registration process of performing registration between the two-dimensional X-ray image and the three-dimensional CT image; and” (Examiner interprets the following as all being part of the registration process. See all of page 3 and page 3 paragraph 7 “ The projection processing unit 3 receives the three-dimensional image data IM1 output from the three-dimensional image input unit 1 and the two-dimensional image data IM2 output from the two-dimensional image input unit 2, and uses IM1 as two-dimensional image data IM3 equivalent to IM2. (Step S400).” Therefore IM3 represents the CT image. See all of page 4, and page 4 paragraphs 3-4, the registration is performed “The registration unit 4 performs registration between the two-dimensional image data IM2 output from the two-dimensional image input unit 2 and the two-dimensional image data IM3 output from the projection processing unit 3, and results in an image after registration. IM2 ′ is output to the composition processing unit 5 and the accuracy evaluation unit 8 (step S500).” See also page 4 paragraph 11. See also page 5 paragraph 1. Makoto)
“a conversion process of converting the first disease label into a two-dimensional second disease label corresponding to the two-dimensional X-ray image on the basis of a result of the registration between the two-dimensional X-ray image and the three-dimensional CT image.” (See all of page 3 and 4, see also page 4 paragraph 11 “The composite image IM4 is obtained by converting the structure data O having position information into a position on the two-dimensional image data IM3, and simply superimposing it on IM2 ′ aligned with IM3 to output the three-dimensional image. The notable structure O extracted in advance by IM1 is mapped and displayed on the two-dimensional image IM2. Here, the structure data O may be colored and displayed for easy viewing, or the transmittance may be set high so that an object behind the structure can be visually recognized.” Examiner interprets the 2D second disease label as the mapped structure 0. Makoto )
Claim 16 is rejected under the same analysis as claim 1.
As per claim 2, Makoto teaches “The disease label creation device according to claim 1, wherein the registration process includes: a process of projecting the CT image to create a pseudo X-ray image; and a process of performing registration between the two-dimensional X-ray image and the pseudo X-ray image. (Examiner interprets IM3 as the pseudo x-ray image. Seer page 3 paragraph 7 “The projection processing unit 3 receives the three-dimensional image data IM1 output from the three-dimensional image input unit 1 and the two-dimensional image data IM2 output from the two-dimensional image input unit 2, and uses IM1 as two-dimensional image data IM3 equivalent to IM2. (Step S400)” See also page 4 paragraphs 3-4 and 11. On paragraph 4 it shows “The registration unit 4 performs registration between the two-dimensional image data IM2 output from the two-dimensional image input unit 2 and the two-dimensional image data IM3 output from the projection processing unit 3, and results in an image after registration.” See also page 5 paragraph 1 “Two-dimensional image data IM1, IM3 obtained by converting IM1 into two-dimensional image data, IM2 aligned with IM3, and composite image IM4 output from the composition processing unit 5 can be selectively displayed and interpreted, or the above It is desirable that a plurality of images can be simultaneously displayed and interpreted, and any output device may be used as long as this purpose can be realized.” Makoto)
As per claim 3, Makoto teaches “The disease label creation device according to claim 1, wherein the registration process includes:
a process of extracting a two-dimensional anatomical landmark from the two-dimensional X-ray image; (Examiner interprets “anatomical landmark” as a region or part of the anatomy. See page 3 paragraphs 1-2 “The two-dimensional image input unit 2 inputs the two-dimensional image data of the subject from an imaging device (not shown) or a file server (step S200). In the present embodiment, the two-dimensional image data is chest X-ray image data obtained by photographing the same subject as the subject displayed in the chest CT image data input from the three-dimensional image input unit 1. In the present embodiment, the subject is the chest of the human body, but is not limited to this, and may be another subject, for example, a blood vessel that has undergone contrast imaging.” See also page 3 paragraph 9 “The imaging conditions of the input 2D image data IM2 are acquired (step S401). Here, the imaging conditions are positional relationship information of the subject, the X-ray tube, and the sensor for obtaining the orientation and imaging region of the subject in IM2.” Makoto.)
a process of extracting a three-dimensional anatomical landmark corresponding to the two-dimensional anatomical landmark from the CT image; (See page 3 paragraph 3 “Next, the information extraction unit 7 extracts a specific structure O displayed in the 3D image data output from the 3D image input unit 1 (step S300). In the present embodiment, the blood vessel structure is extracted from the chest CT image data and output” It corresponds since they are of the same subject. Makoto)
a process of projecting the three-dimensional anatomical landmark; and (See page 3 paragraphs 7-12, para 7 shows “ The projection processing unit 3 receives the three-dimensional image data IM1 output from the three-dimensional image input unit 1 and the two-dimensional image data IM2 output from the two-dimensional image input unit 2, and uses IM1 as two-dimensional image data IM3 equivalent to IM2. (Step S400)” and page 4 paragraphs 1-3, para3 on page 4 shows “By the above processing, the projection processing imitating the photographing means of the two-dimensional image data IM2 can be added to the three-dimensional image data IM1 and converted into the projection image data IM3 that is two-dimensional image data. In the present embodiment, the three-dimensional image IM1 is chest CT image data, and the two-dimensional image data IM2 is chest image data of the same subject, and the chest CT image data is converted into chest image data by projection processing” Makoto)
a process of performing registration between the two-dimensional anatomical landmark and an anatomical landmark after the projection process. (See page 4 paragraph 4 “The registration unit 4 performs registration between the two-dimensional image data IM2 output from the two-dimensional image input unit 2 and the two-dimensional image data IM3 output from the projection processing unit 3, and results in an image after registration.” IM2 contains the 2D anatomical landmark and IM3 contains an anatomical landmark after the projection (it is output). See also page 4 paragraph 8 “The accuracy evaluation unit 8 evaluates the alignment result. In the present embodiment, the image similarity is calculated using the density of the chest rib region, and if it is equal to or less than a certain threshold T, it is determined that sufficient accuracy is not obtained. The position of the projection plane P is finely adjusted, and the projection process is performed again.” Makoto)
As per claim 4, Makoto teaches “The disease label creation device according to claim 1, wherein the registration process includes:
a process of extracting a two-dimensional anatomical region of interest from the two-dimensional X-ray image; (Examiner interprets “anatomical region of interest” as a region or part of the anatomy. See page 3 paragraphs 1-2 “The two-dimensional image input unit 2 inputs the two-dimensional image data of the subject from an imaging device (not shown) or a file server (step S200). In the present embodiment, the two-dimensional image data is chest X-ray image data obtained by photographing the same subject as the subject displayed in the chest CT image data input from the three-dimensional image input unit 1. In the present embodiment, the subject is the chest of the human body, but is not limited to this, and may be another subject, for example, a blood vessel that has undergone contrast imaging.” See also page 3 paragraph 9 “The imaging conditions of the input 2D image data IM2 are acquired (step S401). Here, the imaging conditions are positional relationship information of the subject, the X-ray tube, and the sensor for obtaining the orientation and imaging region of the subject in IM2.” Makoto.)
a process of extracting a three-dimensional anatomical region of interest corresponding to the two-dimensional anatomical region of interest from the CT image; (See page 3 paragraph 3 “Next, the information extraction unit 7 extracts a specific structure O displayed in the 3D image data output from the 3D image input unit 1 (step S300). In the present embodiment, the blood vessel structure is extracted from the chest CT image data and output” It corresponds since they are of the same subject. Makoto)
a process of projecting the three-dimensional anatomical region of interest; and (See page 3 paragraphs 7-12, para 7 shows “ The projection processing unit 3 receives the three-dimensional image data IM1 output from the three-dimensional image input unit 1 and the two-dimensional image data IM2 output from the two-dimensional image input unit 2, and uses IM1 as two-dimensional image data IM3 equivalent to IM2. (Step S400)” and page 4 paragraphs 1-3, para3 on page 4 shows “By the above processing, the projection processing imitating the photographing means of the two-dimensional image data IM2 can be added to the three-dimensional image data IM1 and converted into the projection image data IM3 that is two-dimensional image data. In the present embodiment, the three-dimensional image IM1 is chest CT image data, and the two-dimensional image data IM2 is chest image data of the same subject, and the chest CT image data is converted into chest image data by projection processing” Makoto)
a process of performing registration between a contour of the two-dimensional anatomical region of interest and a contour of an anatomical region of interest after the projection process. (Examiner interprets “contour” as the region of interest. See page 4 paragraph 4 “The registration unit 4 performs registration between the two-dimensional image data IM2 output from the two-dimensional image input unit 2 and the two-dimensional image data IM3 output from the projection processing unit 3, and results in an image after registration.” IM2 contains the 2D anatomical landmark and IM3 contains an anatomical landmark after the projection (it is output). See also page 4 paragraph 8 “The accuracy evaluation unit 8 evaluates the alignment result. In the present embodiment, the image similarity is calculated using the density of the chest rib region, and if it is equal to or less than a certain threshold T, it is determined that sufficient accuracy is not obtained. The position of the projection plane P is finely adjusted, and the projection process is performed again.” See also page 4 paragraphs 11-12 (with the following page) “ The composite image IM4 is obtained by converting the structure data O having position information into a position on the two-dimensional image data IM3, and simply superimposing it on IM2 ′ aligned with IM3 to output the three-dimensional image. The notable structure O extracted in advance by IM1 is mapped and displayed on the two-dimensional image IM2. Here, the structure data O may be colored and displayed for easy viewing, or the transmittance may be set high so that an object behind the structure can be visually recognized. Priorities may be given to the target of interest and displayed in different colors.” A superimposed contour is displayed. Makoto)
As per claim 12, Makoto teaches “The disease label creation device according to claim 1, wherein, in the registration process, the registration is performed by adjusting a solution space in the registration between the two-dimensional X-ray image and the CT image forming the pair associated with a patient, depending on the patient.” (See page 4 paragraph 8, it forms part of the registration “The accuracy evaluation unit 8 evaluates the alignment result. In the present embodiment, the image similarity is calculated using the density of the chest rib region, and if it is equal to or less than a certain threshold T, it is determined that sufficient accuracy is not obtained. The position of the projection plane P is finely adjusted, and the projection process is performed again. The threshold T used for evaluation is obtained experimentally. In addition, a threshold value T may be determined by a doctor who is a user while visually confirming the alignment result.” Examiner interprets “solution space” as any space (pixel/region/image) part of the working solution used for the result. The patient is the same as seen in page 4 paragraphs 2-3 and page 9 paragraph 2 “the three-dimensional image data and the first two-dimensional image data are image data of the same patient taken at different time points.”, it therefore is a pair. Makoto)
As per claim 17, Makoto teaches “a non-transitory, computer-readable tangible recording medium on which a program for causing, when read by a computer, the computer to execute the disease label creation method according to claim 16 is recorded. “ (See page 7 paragraph 10 “ Recording media for supplying the program include the following media. For example, flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), etc. is there.” Makoto)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 6, 7, 8, 10, 18, 19, 21 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Makoto in view of Masubuchi et. el. (US Pub. No. 20200160981-A1).
As per claim 6, Makoto already teaches “the disease label creation device according to claim 1, wherein the first processor is configured to:..” and “second disease label”, however Makoto does not teach “execute a first reliability calculation process of calculating a first reliability for the second disease label.”
Masubuchi teaches “execute a first reliability calculation process of calculating a first reliability for the… disease label.” (Paragraphs 49-55 shows the process of setting a reliability for each label. “[0048]… The reliability setting function 144 can acquire awareness information of a training data creator by receiving a reply to the inquiry of the image S2. When a plurality of labels has been set to medical images, the above-described inquiry may be performed for each label or collectively performed for each medical image.” [0050] “[0050] In the training data 156 with reliability, information of items such as a “medical image,” “label information,” “operation situations” and “creator information” is associated with a “training data ID” that is identification information of training data. ”. See also paragraphs 87 “ In the example of FIG. 10, the acquisition function 242 acquires training data 252 with reliability from each medical institution terminal 100 (step S150). Next, the weighting function 244 sets a weight score to a label set to a medical image on the basis of reliability information of the acquired training data with reliability to generate weighted training data (step S152).” See also figs. 1-4 and 10-11. Masubuchi)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Makoto with the teachings of Masubuchi to calculate a reliability of the second disease label. The modification would have been motivated by the desire to confidently choose correct/reliable information for training with weights, therefore it is an improvement, as suggested by Masubuchi (See paragraphs 51-52 “[0051] The “operation situations” include, for example, information associated with items such as “time required for creation,” “confidence degree,” “freshness degree” and “quality.” The “time required for creation” is, for example, an index indicating a degree of awareness with respect to deficiency and excess of a time required for creation associated with choices of reply information for the first inquiry information of the image S2. The reliability setting function 144 sets a confidence degree based on a reply result for the first inquiry information. ” “[0052] The “confidence degree” is a quality of subjective training data of a training data creator (reliability of trusting that the training data is correct)...” See also paragraph 64-69 “Specifically, the weighting function 244 sets weight scores such that reliability increases as a creation time of a medical image or training data acquired through the acquisition function 242 becomes later. Accordingly, training data that is correct at this point in time can be acquired with high efficiency and training data with stabilized quality is acquired.” Masubuchi)
As per claim 7, Makoto in view of Masubuchi already teaches “the disease label creation device according to claim 6, wherein, in the first reliability calculation process,” and “second disease label”, however Makoto also teaches “a visibility of a second disease region corresponding to the second disease label with respect to a normal region of the two-dimensional X-ray image, the normal region being a region other than the second disease region, is calculated (See page 4 paragraphs 8-9, they show a visibility of the alignment which is used to present the mapped structure data as seen in paragraphs 10-12 “The composite image IM4 is obtained by converting the structure data O having position information into a position on the two-dimensional image data IM3, and simply superimposing it on IM2 ′ aligned with IM3 to output the three-dimensional image. The notable structure O extracted in advance by IM1 is mapped and displayed on the two-dimensional image IM2. Here, the structure data O may be colored and displayed for easy viewing, or the transmittance may be set high so that an object behind the structure can be visually recognized. Priorities may be given to the target of interest and displayed in different colors. This synthesis method may be any method as long as it effectively presents information required by the doctor”. The region includes the processes seen on page 3 paragraphs 8-12. See also page 4 paragraph 8 “In the present embodiment, the image similarity is calculated using the density of the chest rib region, and if it is equal to or less than a certain threshold T…” Examiner interprets “visibility” as any process utilizing pixel values from the images. Examiner also interprets “second disease region” as the region presented by the position information of the mapped structure O. It is with respect to a “normal region” since it is interpreted as the structure data N of IM2 which is used to emphasize structure O as seen on page 6 paragraph 8 “In the present embodiment, the structure data O has three-dimensional position information. Considering this, the structure data N that is three-dimensionally overlapped with the structure data O is obtained in the two dimensional image data IM3. By subtracting the structure data N from the two-dimensional image data IM2, the structure N overlapping the structure data O to be noticed is removed, and the structure data O to be noticed on IM2 is emphasized (see FIG. 9).”. Makoto) using at least one of statistics of pixel values of a normal region and a first disease region of the CT image corresponding to the first disease label or a shape feature of the first disease region of the CT image,” (See page 3 paragraphs 9-12 “The imaging conditions of the input 2D image data IM2 are acquired (step S401). Here, the imaging conditions are positional relationship information of the subject, the X-ray tube, and the sensor for obtaining the orientation and imaging region of the subject in IM2. In the coordinate system of the three-dimensional image data IM1 input using this information, a plane P is constructed with the focal point S (X , s s Y , Z ) at the X-ray tube position and the projection plane at the sensor position. . Here, the plane P has pixel information reflecting the s number of pixels and the pixel pitch of the original sensor, and the pixel of interest (X , Y , Z ) on the plane P is a pixel with the original p sensor. And one-to-one correspondence.” It uses both IM1, IM2 and IM3 information of each region. See also page 4 paragraphs 1-8 “In the present embodiment, alignment of the thoracic bone region with little deformation even in the time-lapse image is performed, and a rigid registration is used as a technique.” Makoto.) “and the first reliability is calculated from the calculated visibility.” (Masubuchi already teaches reliability and it would have been obvious to implement the visibility with reliability at least for the same reason presented in the obviousness rationale. See also paragraph 35. Masubuchi)
As per claim 8, Makoto in view of Masubuchi already teaches “the disease label creation device according to claim 6, wherein, in the information acquisition process, information of an anatomical region in the CT image is acquired, and in the first reliability calculation process” and Masubuchi already teaches “the first reliability is calculated from the calculated visibility” (See paragraphs 35, 48 “The first inquiry information displays choices of “more than sufficient,” “sufficient,” “slightly insufficient” and “insufficient” and the second inquiry information displays choices of “confident,” “normal” and “not confident.” The reliability setting function 144 receives one of those choices from an operator. The reliability setting function 144 can acquire awareness information of a training data creator by receiving a reply to the inquiry of the image S2. When a plurality of labels has been set to medical images, the above-described inquiry may be performed for each label or collectively performed for each medical image. ” and 51 “[0052] The “confidence degree” is a quality of subjective training data of a training data creator (reliability of trusting that the training data is correct). The “confidence degree” is set based on a reply result for the second inquiry information of the image S2. ” Examiner interprets visibility as a quality measure of the image according to a label (which shows the region area) Masubuchi), however Makoto also teaches “a visibility of a second disease region corresponding to the second disease label with respect to a normal region of the two-dimensional X-ray image, the normal region being a region other than the second disease region, is calculated on the basis of superimposition of the anatomical region” and a first disease region of the CT image corresponding to the first disease label in a projection direction, (See page 4 paragraphs 1-11, para 11 shows a visibility of the second region that corresponds to the mapped structure (label) and is according to the x-ray image “ The composite image IM4 is obtained by converting the structure data O having position information into a position on the two-dimensional image data IM3, and simply superimposing it on IM2 ′ aligned with IM3 to output the three-dimensional image. The notable structure O extracted in advance by IM1 is mapped and displayed on the two-dimensional image IM2. Here, the structure data O may be colored and displayed for easy viewing, or the transmittance may be set high so that an object behind the structure can be visually recognized. Priorities may be given to the target of interest and displayed in different colors. This synthesis method may be any method as long as it effectively presents information required by the doctor”. See also page 6 paragraphs 2-3 “However, using the fact that this specific structure data is extracted from the 3D image data as information about the anatomical structure and the diagnosis result, further enhancement processing is added to the composite image to emphasize the specific structure. It may be output as an image.” It is with respect to a “normal region” since it is interpreted as the structure data N of IM2 which is used to emphasize structure O as seen on page 6 paragraph 8 “In the present embodiment, the structure data O has three-dimensional position information. Considering this, the structure data N that is three-dimensionally overlapped with the structure data O is obtained in the two dimensional image data IM3. By subtracting the structure data N from the two-dimensional image data IM2, the structure N overlapping the structure data O to be noticed is removed, and the structure data O to be noticed on IM2 is emphasized (see FIG. 9).”. Makoto).
As per claim 10, Makoto in view of Masubuchi already teaches “the disease label creation device according to claim 6, wherein the first processor is configured to: calculate a degree of success of the result of the registration” (See page 4 paragraphs 8-9 “ The accuracy evaluation unit 8 evaluates the alignment result. In the present embodiment, the image similarity is calculated using the density of the chest rib region, and if it is equal to or less than a certain threshold T, it is determined that sufficient accuracy is not obtained…” “By repeating this series of processes in which the output of the alignment unit 4 is fed back to the projection processing unit 3 via the accuracy evaluation unit 8, the conversion process from the three-dimensional image data IM1 to the two-dimensional image data IM3 is performed. Increases accuracy” Makoto) , and in the first reliability calculation process, the first reliability is calculated on the basis of the degree of success.” (See paragraphs 49-55 and 87. Para 48 shows “The first inquiry information displays choices of “more than sufficient,” “sufficient,” “slightly insufficient” and “insufficient” and the second inquiry information displays choices of “confident,” “normal” and “not confident.” The reliability setting function 144 receives one of those choices from an operator.” 51 shows “The reliability setting function 144 sets a confidence degree based on a reply result for the first inquiry information.” See also figs 1-4 and 10-11. The reliability is based on a degree of success. )
As per claim 18, Makoto already teaches “a two-dimensional X-ray image and the second disease label created by the disease label creation device according to claim 1” , however Makoto does not teach “A learning device comprising: a second processor, wherein the second processor is configured to: execute a learning process of training a disease detection model, using first training data consisting of a two-dimensional x-ray image and the… disease label… and converging a first error between an output of the disease detection model and the… disease label”
Masubuchi teaches “A learning device comprising: a second processor, wherein the second processor is configured to: (See paragraph 33.) execute a learning process of training a disease detection model, using first training data consisting of a two-dimensional x-ray image and the… disease label… (See paragraphs 20, 27, 35 “[0035] For example, the training data creation function 142 may generate training data with respect to a medical image on the basis of an input operation… the training data creation function 142 displays each medical image or every related medical image included in the medical image DB 152 on the display 130 and receives set of labels through the input interface 120. A label includes, for example, at least one indication of the presence or absence of a lesion for a medical image, identification of a lesion type, and designation of a specific region such as a region of interest (ROI). In addition, a label may include identification of a portion of a test object (e.g., a human body) with respect to a medical image, presence or absence of a disease of the portion, designation of a diseased region (e.g., fibrillogenesis of the lung, a solitary pulmonary nodule, or a region of brain tumor) and the like, a numerical value of seriousness of a disease (e.g., a fatty liver level), and the like.” and 126-127 “[0127] According to at least one of the above-described embodiments, it is possible to clarify the quality of training data by including the training data creation function 142 for acquiring training data of a medical image and the reliability setting function 144 for setting, to the training data acquired through the training data creation function 142, reliability information based on one or both of a creation situation of the training data and information on a creator who has created the training data in the medical image processing apparatus. In addition, according to the first embodiment, it is possible to acquire a more appropriate learned model and learning results by adding the quality of training data thereto.”) (See also paragraph 23 “Medical images include computed tomography (CT) magnetic resonance (MR) images, mammography images, endoscopy images, X-ray images, etc…” Masubuchi) converging a first error between an output of the disease detection model and the… second disease label” (See paragraph 39 “[0039]… For example, the “technical score” may be a technical level based on a cumulative number of creations of training data in the past or a technical level derived on the basis of a difference between a label set to a medical image having a correct answer label in advance without teaching the correct answer and the correct answer level. A label difference is, for example, an error of a label set to a medical image and an ROI position difference, presence or absence of a disease, or a difference between disease details. I” Therefore a label difference is within the BRI (broadest reasonable interpretation) of output of the disease detection model (a label showing the disease) and second disease label (the other label used in the difference). See also fig. 8 and paragraphs 76 and 79. See also the loss (error) of paragraphs 79-81 which can also be interpreted as the error. Masubuchi)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Makoto with the teachings of Masubuchi to utilize a learning and training process of the disease along with calculating an error. The modification would have been motivated by the desire to improve the quality and efficiency of data selection/creation, therefore it is an improvement, as suggested by Masubuchi (See paragraphs 3, 106 “[0003]… However, training data items in the medical field are important because they are few in number and are subject to many constraints such as a small number of patients and a necessity of training data to be created under the supervision of doctors. Further, such training data is created at a plurality of sites and by a plurality of doctors in many cases, and the quality of training data collected from a plurality of sources (medical institution terminals, and the like) may not be consistent.” See also paragraph 70 and 81 “[0070] With respect to the “quality,” variations in the quality of training data are expected to occur for training data due to the presence or absence of artifacts, SNR difference, and different image conditions. Accordingly, the weighting function 244 can select learning data in optimal image conditions required for learning with high efficiency by setting a weight score to image quality.” [0081]… it is possible to improve the quality of the learned model 258 by generating the learned model 258 on the basis of the aforementioned weight scores.” Masubuchi)
As per claim 19, Makoto in view of Masubuchi already teaches “using second training data consisting of a two-dimensional X-ray image, the second disease label created by the disease label creation device according to claim 6, and the first reliability”, however Masubuchi also teaches “A learning device comprising: a second processor, wherein the second processor is configured to: in a case in which a learning process of training a disease detection model,…. and converging a first error between an output of the disease detection model and the second disease label is performed, (See paragraphs 20, 27, 35 “[0035] For example, the training data creation function 142 may generate training data with respect to a medical image on the basis of an input operation… and receives set of labels through the input interface 120… a label may include identification of a portion of a test object (e.g., a human body) with respect to a medical image, presence or absence of a disease of the portion, designation of a diseased region (e.g., fibrillogenesis of the lung, a solitary pulmonary nodule, or a region of brain tumor) and the like, a numerical value of seriousness of a disease (e.g., a fatty liver level), and the like.” and 126-127 “[0127] According to at least one of the above-described embodiments, it is possible to clarify the quality of training data by including the training data creation function 142 for acquiring training data of a medical image and the reliability setting function 144 for setting, to the training data acquired throughouttraining data creation function 142, reliability information based on one or both of a creation situation of the training data and information on a creator who has created the training data in the medical image processing apparatus. In addition, according to the first embodiment, it is possible to acquire a more appropriate learned model and learning results by adding the quality of training data thereto.”) (See also paragraph 23 “Medical images include computed tomography (CT) magnetic resonance (MR) images, mammography images, endoscopy images, X-ray images, etc…” Masubuchi). See also paragraph 39 “[0039]… For example, the “technical score” may be a technical level based on a cumulative number of creations of training data in the past or a technical level derived on the basis of a difference between a label set to a medical image having a correct answer label in advance without teaching the correct answer and the correct answer level. A label difference is, for example, an error of a label set to a medical image and an ROI position difference, presence or absence of a disease, or a difference between disease details. I” Therefore a label difference is within the BRI (broadest reasonable interpretation) of output of the disease detection model (a label showing the disease) and second disease label (the other label used in the difference). See also fig. 8 and paragraphs 76 and 79. See also the loss (error) of paragraphs 79-81, which can also be interpreted as an error Masubuchi) execute the learning process of adjusting the first error according to the first reliability to train the disease detection model.” (See paragraphs 79, 81, 107, and 109-116 “the adjustment function 248 determines whether adjustment of correction of the learned model is required (step S350)”. Paragraph 79 shows “0079] The learning function 246 includes, for example, a deep neural network (DNN) using a convolutional neural network (CNN). For example, the learning function 246 causes the DNN to learn a neural network through a machine learning algorithm such as error back propagation.” )
It would have also been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Makoto with the teachings of Masubuchi to utilize a learning and training process of the disease along with calculating an error. The modification would have been motivated by the desire to improve the quality and efficiency of data selection/creation, therefore it is an improvement, as suggested by Masubuchi (See paragraphs 3, 106 “[0003]… However, training data items in the medical field are important because they are few in number and are subject to many constraints such as a small number of patients and a necessity of training data to be created under the supervision of doctors. Further, such training data is created at a plurality of sites and by a plurality of doctors in many cases, and the quality of training data collected from a plurality of sources (medical institution terminals, and the like) may not be consistent.” See also paragraph 70 and 81 “[0070] With respect to the “quality,” variations in the quality of training data are expected to occur for training data due to the presence or absence of artifacts, SNR difference, and different image conditions. Accordingly, the weighting function 244 can select learning data in optimal image conditions required for learning with high efficiency by setting a weight score to image quality.” [0081]… it is possible to improve the quality of the learned model 258 by generating the learned model 258 on the basis of the aforementioned weight scores.” Masubuchi)
As per claim 21, Makoto in view of Masubuchi already teaches “the learning device according to claim 19, wherein the second processor is configured to: execute a learning process of directing the disease detection model to output a disease detection result indicating a disease region included in the two-dimensional X-ray image and a second reliability of the disease detection result. (See paragraphs 35-38, the data creation unit which is used for learning processes obtains labels which contain the disease region from the medical images (x-ray images paragraph 23-25). They are displayed as seen in fig. 2, therefore it is output. The second reliability is output as seen on fig. 5, there are several reliabilities shown. See also paragraphs 80, 78 and 88 “[0088] Next, the learning function 246 generates a learned model 258 which outputs label information for an input of a medical image using the weighted training data (step S154) and stores the generated learned model 258 in the memory 250 (step S156).” See also paragraphs 48, 51, 52, 55, “[0055] In addition, the reliability setting function 144 outputs training data with reliability stored in the memory 150 to the information processing server 200 through the network NW.” The reliability shown in Masubuchi at least contains a second reliability. Masubuchi)
As per claim 24, Makoto in view of Masubuchi already teaches “A disease detection model trained by the learning device according to claim 18,”, however Masubuchi also teaches “wherein the disease detection model receives any two-dimensional X-ray image as an input image, detects a disease label from the input two-dimensional X-ray image, and outputs the disease label.” (See fig. 2, an inputted x-ray image shows the detected disease label and outputs it in the display. See also paragraph 35 “[0035] For example, the training data creation function 142 may generate training data with respect to a medical image on the basis of an input operation received through the input interface 120 in a state in which an input screen including the medical image is displayed on the display 130. Specifically, the training data creation function 142 displays each medical image or every related medical image included in the medical image DB 152 on the display 130 and receives set of labels through the input interface 120. A label includes, for example, at least one indication of the presence or absence of a lesion for a medical image, identification of a lesion type, and designation of a specific region such as a region of interest (ROI). In addition, a label may include identification of a portion of a test object (e.g., a human body) with respect to a medical image, presence or absence of a disease of the portion, designation of a diseased region (e.g., fibrillogenesis of the lung, a solitary pulmonary nodule, or a region of brain tumor) and the like…” See also paragraph 88 “[0088] Next, the learning function 246 generates a learned model 258 which outputs label information for an input of a medical image using the weighted training data (step S154)”. See also paragraph 23-24. Masubuchi)
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Makoto in view of Baka (Baka, Nora, et al. "Oriented Gaussian mixture models for nonrigid 2D/3D coronary artery registration." IEEE transactions on medical imaging 33.5 (2014): 1023-1034.) .
As per claim 13, Makoto already teaches “The disease label creation device according to claim 12” and ”…. corresponding to the two-dimensional X-ray image and the CT image forming the pair; and a process of performing non-rigid registration between the two-dimensional X-ray image and the CT image” (page 4 paragraphs 5-7), however Makoto does not teach “further comprising: a database of a statistical deformation model for each patient feature information item, wherein the registration process includes: a process of selecting a corresponding statistical deformation model from the database on the basis of patient feature information of the patient… performing non-rigid registration between the two-dimensional X-ray image and the CT image using the selected statistical deformation model.”
Baka teaches “further comprising: a database of a statistical deformation model for each patient feature information item, wherein the registration process includes: a process of selecting a corresponding statistical deformation model from the database on the basis of patient feature information of the patient… performing non-rigid registration between the two-dimensional X-ray image and the CT image using the selected statistical deformation model.” (See abstract “The oriented GMM registration achieved a median accuracy of 1.06 mm, with a convergence rate of 81% for nonrigid vessel centerline registration on 12 patient datasets, using a statistical shape model. The method thereby outperformed the iterative closest point algorithm, the GMM registration without orientation, and two recently published methods on 2D/3D coronary artery registration.” See also page 5 section III experimental setup paragraph 1-2 “The test datasets were collected from PCI patients for which both preinterventional 4D computed tomography angiography (CTA) and interventional X-ray angiography (XA) were available.…. The statistical shape model for the coronary arteries at a given cardiac phase was built from 4D CTA datasets. ” Examiner interprets “feature information item” as any information regarding a feature of the patients e.g. “coronary arteries”. See also page 8 subsection D. Nonrigid registration using a SSM. “In this work, we use statistical shape models for parameterizing nonrigid registration. Such deformation parameterization is more restrictive than B-spline or thin-plate-spline (TPS) based registrations, which is advantageous in the ill-defined problem of mono-plane 2D/3D registration”. See also page 5 subsection C, page 6 column 1 paragraphs 1-3 and column 2 paragraphs 3-4 and page 8 subsection B. Baka )
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Makoto with the teachings of Baka to use a statistical deformation model when using non-rigid registration. The modification would have been motivated by the desire to have better performance, therefore it is an improvement, as suggested by Baka ( See page 8 column 1 paragraphs 3-4 “Fig. 6 shows the SSM based simulation experiment results. The best median results were achieved by OGMM_.5, DT, and G1, being 0.48 mm, 0.50 mm, and 0.52 mm, respectively. OGMM_.5 and DT were not significantly different. ICP was significantly worse than all others (all )… Fig. 10 shows the results on all frames of all patients for the SSM matching on real data. Unlike the simulation experiments, on real data OGMM_.5 andOGMM_.3 performed significantly different ( ). OGMM_.3 performed best with 81% convergence, and median accuracy of 1.06 mm. It thereby out performed all other methods .” See also abstract “The oriented GMM registration achieved a median accuracy of 1.06 mm, with a convergence rate of 81% for nonrigid vessel centerline registration on 12 patient datasets, using a statistical shape model. The method thereby outperformed the iterative closest point algorithm, the GMM registration without orientation, and two recently published methods on 2D/3D coronary artery registration” Baka)
Claims 14 is rejected under 35 U.S.C. 103 as being unpatentable over Makoto in view of Aoyagi (US Pub. No. 2020/0380680 A1) .
As per claim 14, Makoto already teaches “the disease label creation device according to claim 1, wherein, in the information acquisition process,”, however Makoto does not teach “an image-level third disease label of the CT image is acquired, and the first processor is configured to: give the second disease label and the third disease label to the two-dimensional X-ray image.” Along with CT image and X-ray image.
Aoyagi teaches “an image-level third disease label… is acquired, and the first processor is configured to: give the second disease label and the third disease label to the… image” (See paragraph 45 “Here, the “virtual projection image” is, for example, an image simulating a 2D general X-ray image.” See fig. 4, multiple labels (name and region) are given to the 2d x-ray image. See also paragraphs 46-52. Para 51 shows “[0051] A labeled virtual projection image is illustrated on the upper right side of FIG. 4. In this labeling, the size of the heart observed in the virtual projection image is larger than the normal size, and thus, the label “cardiomegalia” (i.e., cardiac hypertrophy) is attached as the diagnosed disease name. In addition, a label (white broken line) indicating the position of the enlarged heart is attached in the virtual projection image. ” Aoyagi )
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Makoto with the teachings of Aoyagi to apply multiple labels to an image. The modification would have been motivated by the desire to associate correct data, therefore it is an improvement, as suggested by Aoyagi (See paragraph 49 “[0049] In this specification, the verb of “to label” is used for “to associate correct information/data (i.e., ground truth) with the data for training” such as a training image. Further, the noun of “label”, per se, may refer to the correct information/data (i.e., the ground truth).”)
Allowable Subject Matter
Claims 5, 9, 11, 15, 20, 22 and 23 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN J MENDEZ MUNIZ whose telephone number is (703)756-5672. The examiner can normally be reached M-F, 8AM - 5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN JOHN MENDEZ MUNIZ/Examiner, Art Unit 2675
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights