DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
1. The information disclosure statement (IDS) submitted on 7/3/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement has been considered by the examiner.
Claim Rejections - 35 USC § 112
2. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
3. Claim 16 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 16 recites the following: “The method of claim 3, wherein the human user has a perception rate of at most 25 images per second.” The limitation in this claim is attributed to a human user’s perception rate and not the disclosed invention. It is unclear how the declaration of a human perception rate limits the method of independent claim 1 and thus is considered indefinite.
Claim Rejections - 35 USC § 101
4. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
5. Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claim 15 recites the following: “A computer program product comprising a computer program configured to be loaded directly into a memory of a provision unit, …”. The claim does not recite any other structural limitations. The scope of a computer program product comprising a computer program includes software per se. While the claim further recites the program as configured to be loaded, this is only a limitation on the configuration of the program as capable of being loaded, and not a limitation indicating the program as actually loaded. Software per se constitutes subject matter not eligible for patent, and a claim that includes in its scope embodiments not eligible for patent is rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 102
6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
7. Claims 1-2, 5, 8-9, 13-15, and 18 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Yan et al. (WO-2022/051977-A1, hereinafter "Yan").
8. As per claim 1, Yan discloses: A method for alternately displaying graphical representations, the method comprising:
a) providing a first image that has a mapping of an anatomical structure of an object under examination; (Yan, page 2, ¶ 1-4, [0007]-[0010] “On the one hand, this application provides an image registration method, the image registration method comprising: Acquire the first and second medical images of the target object to be registered; A first target observation image is output to a display device based on the first medical image and the second medical image, so as to alternately display the first medical image and the second medical image on the display device; Based on the first target observation image, image registration is performed on the target object.” and page 13, ¶ 8, [0156], “In step 201, the target object can be a tissue or organ of the human body, such as the head, or organs such as the lungs or liver. The first medical image and the second medical image are both medical images of the target object, that is, medical images of the same tissue or organ, such as medical images of the user's brain.”)
b) providing a second image that maps a spatial change and/or time-related change in the object under examination;
c) displaying a graphical representation of the first image or second image on a display surface of a display unit; and (Yan, page 30, ¶ 5-7, [0011]-[0013], “In some embodiments of this application, the step of outputting a first target observation image to a display device based on the first medical image and the second medical image, so as to alternately display the first medical image and the second medical image on the display device, includes: Obtain the preset image output time interval; According to the image output time interval, the first medical image and the second medical image are sequentially used as the first target observation image and output to the display device so as to alternately display the first medical image and the second medical image on the display device.” and page 14, ¶ 7, [0163], “The first medical image and the second medical image are displayed alternately according to the image output time interval. The operator can adjust the position of the first medical image and/or the second medical image while observing the displayed alternating images (adjustment and display are almost real-time) until the first medical image and the second medical image are completely aligned.”)
d) displaying a graphical representation of the respective other image on the display surface of the display unit, wherein acts c) and d) are carried out one after the other in terms of time, and wherein acts b) to d) are repeated. (Yan, page 9, ¶ 13, [0124], “The display module is used to display the image registration interface for target object registration. After receiving the target observation image input by the electronic device, the first medical image and the second medical image of the target object to be registered are alternately displayed on the image registration interface at a preset time interval. The target observation image is an image output by the electronic device based on the first medical image and the second medical image.” and page 14, ¶ 7, [0163], “The first medical image and the second medical image are displayed alternately according to the image output time interval. The operator can adjust the position of the first medical image and/or the second medical image while observing the displayed alternating images (adjustment and display are almost real-time) until the first medical image and the second medical image are completely aligned.”)
9. As per claim 2, Yan discloses: The method of claim 1, wherein the graphical representation in act c) is displayed with a first display duration and the graphical representation in act d) is displayed with a second display duration. (Yan, page 15, ¶ 2-4, [0164]-[0166], “The image output time interval is the time interval between switching between the first medical image and the second medical image. The image output time interval can be preset as needed. The interval time can be 0.25 seconds, 0.5 seconds or 1 second, etc. It can also be left unset or set to a default value, such as 0.25 seconds. In one embodiment of setting the image output time interval in this application, an "automatic switching" button can be set on the image registration interface of the display device. For example, it can be set after the slider for adjusting the transparency of the medical image. When the "automatic switching" button is triggered, a drop-down list is displayed. The drop-down list displays multiple options for selectable image output interval times, such as 0.25 seconds, 0.5 seconds, or 1 second. For example, after the operator selects the interval time, the first target observation image under that image output interval will start to be displayed in the image registration interface. In another approach, an "automatic switching" checkbox can be set in the image registration interface of the display device. Once checked, the first target observation image will begin to be displayed in the image registration interface. The image output time interval can be a default value or can be set by adjusting the progress bar of "Switch Time" or the "Switch Time Setting" button. It should be noted that in this embodiment, the interface display method for setting the image output time interval can be varied and is not limited.” and page 9, ¶ 13, [0124], “The display module is used to display the image registration interface for target object registration. After receiving the target observation image input by the electronic device, the first medical image and the second medical image of the target object to be registered are alternately displayed on the image registration interface at a preset time interval. The target observation image is an image output by the electronic device based on the first medical image and the second medical image.”)
10. As per claim 5, Yan discloses: The method of claim 2, wherein a first user input is recorded, and wherein the first display duration and/or the second display duration is adjusted based on the first user input. (Yan, page 15, ¶ 2-4, [0164]-[0166], “The image output time interval is the time interval between switching between the first medical image and the second medical image. The image output time interval can be preset as needed. The interval time can be 0.25 seconds, 0.5 seconds or 1 second, etc. It can also be left unset or set to a default value, such as 0.25 seconds. In one embodiment of setting the image output time interval in this application, an "automatic switching" button can be set on the image registration interface of the display device. For example, it can be set after the slider for adjusting the transparency of the medical image. When the "automatic switching" button is triggered, a drop-down list is displayed. The drop-down list displays multiple options for selectable image output interval times, such as 0.25 seconds, 0.5 seconds, or 1 second. For example, after the operator selects the interval time, the first target observation image under that image output interval will start to be displayed in the image registration interface. In another approach, an "automatic switching" checkbox can be set in the image registration interface of the display device. Once checked, the first target observation image will begin to be displayed in the image registration interface. The image output time interval can be a default value or can be set by adjusting the progress bar of "Switch Time" or the "Switch Time Setting" button. It should be noted that in this embodiment, the interface display method for setting the image output time interval can be varied and is not limited.”)
11. As per claim 8, Yan discloses: The method of claim 1, wherein the first image and the second image are registered with each other. (page 2, ¶ 1-4, [0007]-[0010], “On the one hand, this application provides an image registration method, the image registration method comprising: Acquire the first and second medical images of the target object to be registered; A first target observation image is output to a display device based on the first medical image and the second medical image, so as to alternately display the first medical image and the second medical image on the display device; Based on the first target observation image, image registration is performed on the target object.”)
12. As per claim 9, Yan discloses: The method of claim 1, further comprising:
recording a further user input, wherein the further user input determines how often act c) is executed during the repeated execution of acts b) to d). (Yan, page 15, ¶ 2-4, [0164]-[0166], “The image output time interval is the time interval between switching between the first medical image and the second medical image. The image output time interval can be preset as needed. The interval time can be 0.25 seconds, 0.5 seconds or 1 second, etc. It can also be left unset or set to a default value, such as 0.25 seconds. In one embodiment of setting the image output time interval in this application, an "automatic switching" button can be set on the image registration interface of the display device. For example, it can be set after the slider for adjusting the transparency of the medical image. When the "automatic switching" button is triggered, a drop-down list is displayed. The drop-down list displays multiple options for selectable image output interval times, such as 0.25 seconds, 0.5 seconds, or 1 second. For example, after the operator selects the interval time, the first target observation image under that image output interval will start to be displayed in the image registration interface. In another approach, an "automatic switching" checkbox can be set in the image registration interface of the display device. Once checked, the first target observation image will begin to be displayed in the image registration interface. The image output time interval can be a default value or can be set by adjusting the progress bar of "Switch Time" or the "Switch Time Setting" button. It should be noted that in this embodiment, the interface display method for setting the image output time interval can be varied and is not limited.” and page 9, ¶ 13, [0124], “The display module is used to display the image registration interface for target object registration. After receiving the target observation image input by the electronic device, the first medical image and the second medical image of the target object to be registered are alternately displayed on the image registration interface at a preset time interval. The target observation image is an image output by the electronic device based on the first medical image and the second medical image.”)
13. As per claim 13, Yan discloses: The method of claim 1, wherein the first image and the second image are acquired by a same medical imaging device. (Yan, page 12, ¶ 2, [0145], “In this embodiment, the image acquisition device 100 may be a CT device, a CBCT device, or other medical imaging devices, such as ultrasound devices (e.g., B-ultrasound devices or color Doppler ultrasound devices), magnetic resonance imaging (MRI) devices, etc., and is not specifically limited here. In a specific application scenario, the first medical image is the image generated by CBCT in a radiotherapy device, and the second medical image can be the image generated by a CT device, CBCT device, or other medical imaging devices (such as ultrasound devices or MRI devices).” and page 12, ¶ 4, [0147], “For example, Figure 1 only shows one electronic device and one image acquisition device. It is understood that the image registration system may also include one or more other electronic devices, or/and one or more other image acquisition devices connected to the electronic devices via a network. Specifically, for example, one electronic device 200 may connect to multiple image acquisition devices 100, that is, one electronic device 200 may display medical images acquired by multiple image acquisition devices 100, or one image acquisition device 100 may connect to multiple electronic devices 200, that is, the image acquired by one image acquisition device 100 may be output to multiple electronic devices 200 for display. The specifics are not limited here.”)
14. As per claim 14, Yan discloses: A system comprising:
a medical imaging device; and
a display unit having a display surface, wherein the medical imaging device is configured to:
provide a first image that maps an anatomical structure of an object under examination; and (Yan, page 2, ¶ 1-4, [0007]-[0010] “On the one hand, this application provides an image registration method, the image registration method comprising: Acquire the first and second medical images of the target object to be registered; A first target observation image is output to a display device based on the first medical image and the second medical image, so as to alternately display the first medical image and the second medical image on the display device; Based on the first target observation image, image registration is performed on the target object.” and page 12, ¶ 2, [0145], “In this embodiment, the image acquisition device 100 may be a CT device, a CBCT device, or other medical imaging devices, such as ultrasound devices (e.g., B-ultrasound devices or color Doppler ultrasound devices), magnetic resonance imaging (MRI) devices, etc., and is not specifically limited here. In a specific application scenario, the first medical image is the image generated by CBCT in a radiotherapy device, and the second medical image can be the image generated by a CT device, CBCT device, or other medical imaging devices (such as ultrasound devices or MRI devices).” and page 13, ¶ 8, [0156], “In step 201, the target object can be a tissue or organ of the human body, such as the head, or organs such as the lungs or liver. The first medical image and the second medical image are both medical images of the target object, that is, medical images of the same tissue or organ, such as medical images of the user's brain.”)
repeatedly provide a second image mapping a spatial change and/or time-related change in the object under examination, and wherein the display unit is configured to repeatedly display a graphical representation of the first image and a graphical representation of a most recently provided second image on the display surface in chronological order. (Yan, page 9, ¶ 13, [0124], “The display module is used to display the image registration interface for target object registration. After receiving the target observation image input by the electronic device, the first medical image and the second medical image of the target object to be registered are alternately displayed on the image registration interface at a preset time interval. The target observation image is an image output by the electronic device based on the first medical image and the second medical image.” and page 14, ¶ 7, [0163], “The first medical image and the second medical image are displayed alternately according to the image output time interval. The operator can adjust the position of the first medical image and/or the second medical image while observing the displayed alternating images (adjustment and display are almost real-time) until the first medical image and the second medical image are completely aligned.”)
15. Claim 15 is similar in scope to claim 14 except for additional limitations that Yan discloses: A computer program product comprising a computer program configured to be loaded directly into a memory of a provision unit, wherein the computer program, when executed by the provision unit, is configured to: (Yan, page 30, ¶ 4-5, [0306]-[0307], “Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. It stores a computer program, which is loaded by a processor to execute the steps in any of the image registration methods provided in the embodiments of this application. For example, the computer program, when loaded by the processor, can perform the following steps: Acquire the first and second medical images of the target object to be registered ...”)
16. As per claim 18, Yan discloses: The method of claim 1, wherein the first image and the second image are acquired by different medical imaging devices. (Yan, page 12, ¶ 2, [0145], “In this embodiment, the image acquisition device 100 may be a CT device, a CBCT device, or other medical imaging devices, such as ultrasound devices (e.g., B-ultrasound devices or color Doppler ultrasound devices), magnetic resonance imaging (MRI) devices, etc., and is not specifically limited here. In a specific application scenario, the first medical image is the image generated by CBCT in a radiotherapy device, and the second medical image can be the image generated by a CT device, CBCT device, or other medical imaging devices (such as ultrasound devices or MRI devices).” and page 12, ¶ 4, [0147], “For example, Figure 1 only shows one electronic device and one image acquisition device. It is understood that the image registration system may also include one or more other electronic devices, or/and one or more other image acquisition devices connected to the electronic devices via a network. Specifically, for example, one electronic device 200 may connect to multiple image acquisition devices 100, that is, one electronic device 200 may display medical images acquired by multiple image acquisition devices 100, or one image acquisition device 100 may connect to multiple electronic devices 200, that is, the image acquired by one image acquisition device 100 may be output to multiple electronic devices 200 for display. The specifics are not limited here.”)
Claim Rejections - 35 USC § 103
17. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
18. Claims 3-4 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Yan et al. (WO-2022/051977-A1, hereinafter "Yan") in view of Lee et al. (NPL: "Fusion of coregistered cross-modality images using a temporally alternating display method," Med. Biol. Eng. Comput., vol. 38, no. 2, pp. 127–132, Mar. 2000., hereinafter "Lee").
19. As per claim 3, Yan discloses: The method of claim 2, wherein a repetition rate of the repeated execution of act b) to d) and/or the first display duration and/or the second display duration are adjusted such that the graphical representations displayed in acts c) and d) [[are not temporally separable for a human user.]] (See rejection for claim 2.)
20. Yan doesn't explicitly disclose but Lee discloses: [[The method of claim 2, wherein a repetition rate of the repeated execution of act b) to d) and/or the first display duration and/or the second display duration are adjusted such that the graphical representations displayed in acts c) and d)]] are not temporally separable for a human user. (Lee, page 127, ¶ 4, “In this study, we propose a new method for the simultaneous display of coregistered CT/MR and PET images. The basic idea is simple: a rapidly alternating display of both images with independent colour scales allows the fusion of images in the human visual perception system. In addition, a particular aspect of the fused image can be emphasised by increasing the display time and/or the intensity of one image with respect to the other.” and page 129, ¶ 8-9, “In this study, we propose a new strategy where coregistered images can be simultaneously visualised. Rapid alternation of the screen on which each image is displayed with independent colour scales causes the fusion of both images but no loss of spatial resolution. Using Bloch's law, we can explain how the visual system accomplishes this temporal summation or fusion of two images (HALLETT, 1991).” and Lee, page 128, ¶ 9, “To induce optical fusion such that both images were overlaid, the method above was applied to rapid switching of the images. The procedure is demonstrated graphically in Fig. 1. ... The temporal modulation rate for the images was 87 Hz, as we synchronised the display of images with the vertical scanning.” and page 130, ¶ 1, “Temporal sensitivity of the human visual system therefore has a finite limitation. However, it is hard to define the frequency threshold above which the eye does not discern the flickering effect as temporal sensitivity of the human visual system is dependent on the mean background intensity. In this system, the highest detectable temporal frequency ranges from 30-40 Hz at low ambient light levels to 80 Hz at high ambient light levels (WANDELL, 1995; WATSON et al., 1986; FARREL, 1991).”)
21. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 2 of Yan to include the disclosure of adjusting the display duration of the first and second images so that they are not temporally separable for a human user, of Lee. The motivation for this modification could have been to adjust the alternating image frame rate so that the two images appear to be superimposed on each other by the human user. This can provide a user with useful additional information about the object under examination so that the user doesn’t have to look at two different displays. Such a technique also requires little computer processing power as it is dependent on the frame frequency of the hardware display.
22. As per claim 4, Yan in view of Lee discloses: The method of claim 3, wherein an image repetition rate of display of the graphical representations in acts c) and d) is at least 50 images per second. (Lee, page 128, ¶ 9, “To induce optical fusion such that both images were overlaid, the method above was applied to rapid switching of the images. The procedure is demonstrated graphically in Fig. 1. ... The temporal modulation rate for the images was 87 Hz, as we synchronised the display of images with the vertical scanning.”)
23. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 3 of Yan to include the disclosure of
displaying the alternating images at a repetition rate of at least 50 images per second, of Lee. The motivation for this modification could have been to not only make it appear that the images are not temporally separable, but any additional images per second will further make any spatial and/or time changes on the display appear smoother and reduce the appearance of display flicker.
24. As per claim 16, Yan in view of Lee discloses: The method of claim 3, wherein the human user has a perception rate of at most 25 images per second. (Lee, page 128, ¶ 9, “To induce optical fusion such that both images were overlaid, the method above was applied to rapid switching of the images. The procedure is demonstrated graphically in Fig. 1. ... The temporal modulation rate for the images was 87 Hz, as we synchronised the display of images with the vertical scanning.” and page 130, ¶ 1, “Temporal sensitivity of the human visual system therefore has a finite limitation. However, it is hard to define the frequency threshold above which the eye does not discern the flickering effect as temporal sensitivity of the human visual system is dependent on the mean background intensity. In this system, the highest detectable temporal frequency ranges from 30-40 Hz at low ambient light levels to 80 Hz at high ambient light levels (WANDELL, 1995; WATSON et al., 1986; FARREL, 1991).”)
25. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 3 of Yan to include the disclosure of a user with a perception rate of at most 25 images per second, of Lee. The motivation for this modification could have been to ensure that a user with a given perception rate is properly suited to view the alternating images from the display method. Ideally, the user would be able to view the images so that they are not temporally separable.
26. Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Yan et al. (WO-2022/051977-A1, hereinafter "Yan") in view of Sato et al. (US-2022/0151474-A1, hereinafter "Sato").
27. As per claim 6, Yan discloses: The method of claim 1, wherein the spatial change and/or the time-related change [[comprises a movement of a medical object and/or of a contrast agent in the object under examination.]] (See rejection for claim 1.)
28. Yan doesn't explicitly disclose but Sato discloses: [[The method of claim 1, wherein the spatial change and/or the time-related change]] comprises a movement of a medical object and/or of a contrast agent in the object under examination. (Sato, Figure 5; [0050], “Further, the image sensor 522 performs imaging every first and second periods, which are alternately repeated, in synchronization with light emission timings of the light source device 3 under the control of the control device 9. Hereinafter, for convenience of the description, an image generated by capturing the subject image (normal light) during the first period by the image sensor 522 will be referred to as a normal light image, and an image generated by capturing the fluorescent image (fluorescence) during the second period by the image sensor 522 will be referred to as a fluorescence image. In addition, the normal light image and the fluorescence image are collectively referred to as a captured image.” and [0083], “After Step S1, the imaging controller 942 causes the image sensor 522 to capture a subject image and a fluorescent image in the first and second periods in synchronization with light emission timings of the first and second light sources 31 and 32 based on the synchronization signal (Steps S2 to S4). That is, during the first period (Step S2: Yes), in other words, when the inside of a living body is irradiated with normal light (white light), the image sensor 522 captures the subject image (normal light) to generate a normal light image (Step S3). On the other hand, during the second period (Step S2: No), in other words, when the inside of the living body is irradiated with near-infrared excitation light, the image sensor 522 captures a fluorescent image (fluorescence) to generate a fluorescence image (Step S4).” and [0072]-[0074], “The first superimposition process is a process of replacing an area at the same pixel position as a fluorescence area in a normal light image with an image of the fluorescence area in a fluorescence image. The second superimposition process is a process (so-called alpha blend process) of changing brightness of a color indicating fluorescence applied to each pixel in an area at the same pixel position as a fluorescence area in a normal light image according to a luminance value at each pixel position in the fluorescence area of a fluorescence image. The display controller 935 generates a video signal for displaying the superimposed image generated by the superimposed image generation unit 934 under the control of the control unit 94. Further, the display controller 935 outputs the video signal to the display device 7 via the second transmission cable 8.” and [0027], “The camera head 5 corresponds to an imaging device according to the present disclosure.” and [0104], “For example, a configuration may be adopted in which picture-in-picture processing or the like is executed, and the normal light image and the fluorescence image are simultaneously displayed on the display device 7.”; Examiner’s note: Although not explicitly stated, fluorescent images are generated through the use of a contrast agent. In addition, the configuration of adopting a “picture-in-picture” to simultaneously display both a normal light image and fluorescent image includes alternately displaying the images, as it is a subset of displaying all images.)
29. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Yan to include the disclosure of the spatial and/or time changes in the images consist of either the movement of a medical object and/or a contrast agent, of Sato. The motivation for this modification could have been to ensure that the alternating image display method is focused on the medical object and/or contrast agent for the benefit of the user.
30. As per claim 7, Yan in view of Sato discloses: The method of claim 6, wherein the anatomical structure comprises a hollow organ, and wherein the second image maps the spatial change and/or the time-related change in the hollow organ. (Yan, page 13, ¶ 8, [0156], “In step 201, the target object can be a tissue or organ of the human body, such as the head, or organs such as the lungs or liver. The first medical image and the second medical image are both medical images of the target object, that is, medical images of the same tissue or organ, such as medical images of the user's brain.” and Yan, page 14, ¶ 7, [0163], “The first medical image and the second medical image are displayed alternately according to the image output time interval. The operator can adjust the position of the first medical image and/or the second medical image while observing the displayed alternating images (adjustment and display are almost real-time) until the first medical image and the second medical image are completely aligned.” and Yan, page 2, ¶ 1-4, [0007]-[0010], “On the one hand, this application provides an image registration method, the image registration method comprising: Acquire the first and second medical images of the target object to be registered; A first target observation image is output to a display device based on the first medical image and the second medical image, so as to alternately display the first medical image and the second medical image on the display device; Based on the first target observation image, image registration is performed on the target object.” and Sato, Figure 5; [0083], “After Step S1, the imaging controller 942 causes the image sensor 522 to capture a subject image and a fluorescent image in the first and second periods in synchronization with light emission timings of the first and second light sources 31 and 32 based on the synchronization signal (Steps S2 to S4). That is, during the first period (Step S2: Yes), in other words, when the inside of a living body is irradiated with normal light (white light), the image sensor 522 captures the subject image (normal light) to generate a normal light image (Step S3). On the other hand, during the second period (Step S2: No), in other words, when the inside of the living body is irradiated with near-infrared excitation light, the image sensor 522 captures a fluorescent image (fluorescence) to generate a fluorescence image (Step S4).” And Sato, [0072]-[0074], “The first superimposition process is a process of replacing an area at the same pixel position as a fluorescence area in a normal light image with an image of the fluorescence area in a fluorescence image. The second superimposition process is a process (so-called alpha blend process) of changing brightness of a color indicating fluorescence applied to each pixel in an area at the same pixel position as a fluorescence area in a normal light image according to a luminance value at each pixel position in the fluorescence area of a fluorescence image. The display controller 935 generates a video signal for displaying the superimposed image generated by the superimposed image generation unit 934 under the control of the control unit 94. Further, the display controller 935 outputs the video signal to the display device 7 via the second transmission cable 8.” and Sato, [0027], “The camera head 5 corresponds to an imaging device according to the present disclosure.”)
31. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 6 of Yan to include the disclosure of the second image mapping the spatial and/or time change within the anatomical structure of a hollow organ, of Sato. The motivation for this modification could have been to assist a user in viewing the changes within a hollow organ while under examination.
32. Claims 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Yan et al. (WO-2022/051977-A1, hereinafter "Yan") in view of Novak (US-2006/0171578-A1).
33. As per claim 10, Yan discloses: The method of claim 1, wherein the providing of the first image [[comprises acquisition of a first mask image and a first fill image, and wherein the first image is generated from the first mask image and the first fill image as a difference image.]] (See rejection for claim 1.)
34. Yan doesn't explicitly disclose but Novak discloses: [[The method of claim 1, wherein the providing of the first image]] comprises acquisition of a first mask image and a first fill image, and wherein the first image is generated from the first mask image and the first fill image as a difference image. (Novak, Figure 5; [0010]-[0011], “In one embodiment of the present invention, a method for splicing medical image datasets comprises: segmenting first and second medical image datasets comprising an organ of interest and a nearby area to create a mask for points in the first and second medical image datasets, wherein the mask identifies points in the organ of interest and nearby area; and creating a spliced image of the first and second medical image datasets by using the points in the organ of interest and nearby area identified by the mask. The first and second medical image datasets are segmented by an automatic or semi-automatic segmentation technique. The first and second medical image datasets are acquired simultaneously or at different times.” and [0058]-[0060], “Once the segmentation of the image data has been completed, a mask specifying each point in the image dataset of the image data is created. In other words, a mask that identifies which pixels or voxels of the acquired image datasets are part of the organ or organs of interest is generated by the segmentation. In more detail, once an organ of interest has been segmented, a mask m is provided such that:
m[i,j] = 1 for pixels within the organ
m[i,j] = 0 for all other pixels [1]
where i and j specify indices in the image. … Given the mask, a new composite or spliced image is created by using the pixels from the image datasets (230). This is accomplished by first assuming (in this example) that there are two datasets for a body part of interest, which may contain two or more organs of interest. In other words, the image data of the body part includes a first CT image dataset and a second CT image dataset. The datasets are defined as, D1[i,j] and D2[i,j], where D1 is a dataset intended for viewing the organ of interest and D2[i,j] is a dataset intended for viewing a different organ that falls within the same region of the body. For example, D1 may be intended for viewing the lungs, whereas D2 may be intended for viewing the soft tissues of the chest. In this case, the spliced image S will be given as:
S[i,j] = D1[i,j] * m[i,j] + D2[i,j] * (1−m[i,j]) [2]
As shown by equation [2], since m takes on only values of 0 or 1, either the first half of the equation will have a value of zero, in which case the pixels from the second dataset are spliced into the image S, or the second half of the equation will have a value of zero, in which case the pixels from the first dataset will be spliced into the image S.” and [0061]-[0062], “If multiple organs of interest have been extracted, a mask for each organ may be provided. For example, if multiple organs have been extracted from a CT scan of a patient's chest, there will be a mask specifying the location of the lungs and a mask specifying a location of the ribs. ... In addition, there may be a default mask mn, which, for example, is intended to view all structures outside the extracted organ or organs ...”)
35. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Yan to include the disclosure of generating the first image as a difference image from a first mask image and a first fill image, of Novak. The motivation for this modification could have been to use the mask and fill images to focus on a region(s) or area(s) of interest and filter out any unnecessary noise or objects in the images. This will provide for clearer images, making it easier for a user to focus on a region while under examination.
36. As per claim 11, Yan in view of Novak discloses: The method of claim 10, wherein the providing of the second image comprises acquisition of a second fill image, and wherein the second image is generated from the first mask image and the second fill image as a difference image. (Novak, Figure 5; [0010]-[0011], “In one embodiment of the present invention, a method for splicing medical image datasets comprises: segmenting first and second medical image datasets comprising an organ of interest and a nearby area to create a mask for points in the first and second medical image datasets, wherein the mask identifies points in the organ of interest and nearby area; and creating a spliced image of the first and second medical image datasets by using the points in the organ of interest and nearby area identified by the mask. The first and second medical image datasets are segmented by an automatic or semi-automatic segmentation technique. The first and second medical image datasets are acquired simultaneously or at different times.” and [0058]-[0060], “Once the segmentation of the image data has been completed, a mask specifying each point in the image dataset of the image data is created. In other words, a mask that identifies which pixels or voxels of the acquired image datasets are part of the organ or organs of interest is generated by the segmentation. In more detail, once an organ of interest has been segmented, a mask m is provided such that:
m[i,j] = 1 for pixels within the organ
m[i,j] = 0 for all other pixels [1]
where i and j specify indices in the image. … Given the mask, a new composite or spliced image is created by using the pixels from the image datasets (230). This is accomplished by first assuming (in this example) that there are two datasets for a body part of interest, which may contain two or more organs of interest. In other words, the image data of the body part includes a first CT image dataset and a second CT image dataset. The datasets are defined as, D1[i,j] and D2[i,j], where D1 is a dataset intended for viewing the organ of interest and D2[i,j] is a dataset intended for viewing a different organ that falls within the same region of the body. For example, D1 may be intended for viewing the lungs, whereas D2 may be intended for viewing the soft tissues of the chest. In this case, the spliced image S will be given as:
S[i,j] = D1[i,j] * m[i,j] + D2[i,j] * (1−m[i,j]) [2]
As shown by equation [2], since m takes on only values of 0 or 1, either the first half of the equation will have a value of zero, in which case the pixels from the second dataset are spliced into the image S, or the second half of the equation will have a value of zero, in which case the pixels from the first dataset will be spliced into the image S.” and [0061]-[0062], “If multiple organs of interest have been extracted, a mask for each organ may be provided. For example, if multiple organs have been extracted from a CT scan of a patient's chest, there will be a mask specifying the location of the lungs and a mask specifying a location of the ribs. ... In addition, there may be a default mask mn, which, for example, is intended to view all structures outside the extracted organ or organs ...”)
37. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 10 of Yan to include the disclosure of generating the second image as a difference image from the first mask image and the second fill image, of Novak. The motivation for this modification could have been to use the mask and fill images to focus on a region(s) or area(s) of interest and filter out any unnecessary noise or objects in the images. This will provide for clearer images, making it easier for a user to focus on a region while under examination.
38. As per claim 12, Yan in view of Novak discloses: The method of claim 10, wherein the providing of the second image comprises acquisition of a second mask image and a second fill image, and wherein the second image is generated from the second mask image and the second fill image as a difference image. (Novak, Figure 5; [0010]-[0011], “In one embodiment of the present invention, a method for splicing medical image datasets comprises: segmenting first and second medical image datasets comprising an organ of interest and a nearby area to create a mask for points in the first and second medical image datasets, wherein the mask identifies points in the organ of interest and nearby area; and creating a spliced image of the first and second medical image datasets by using the points in the organ of interest and nearby area identified by the mask. The first and second medical image datasets are segmented by an automatic or semi-automatic segmentation technique. The first and second medical image datasets are acquired simultaneously or at different times.” and [0058]-[0060], “Once the segmentation of the image data has been completed, a mask specifying each point in the image dataset of the image data is created. In other words, a mask that identifies which pixels or voxels of the acquired image datasets are part of the organ or organs of interest is generated by the segmentation. In more detail, once an organ of interest has been segmented, a mask m is provided such that:
m[i,j] = 1 for pixels within the organ
m[i,j] = 0 for all other pixels [1]
where i and j specify indices in the image. … Given the mask, a new composite or spliced image is created by using the pixels from the image datasets (230). This is accomplished by first assuming (in this example) that there are two datasets for a body part of interest, which may contain two or more organs of interest. In other words, the image data of the body part includes a first CT image dataset and a second CT image dataset. The datasets are defined as, D1[i,j] and D2[i,j], where D1 is a dataset intended for viewing the organ of interest and D2[i,j] is a dataset intended for viewing a different organ that falls within the same region of the body. For example, D1 may be intended for viewing the lungs, whereas D2 may be intended for viewing the soft tissues of the chest. In this case, the spliced image S will be given as:
S[i,j] = D1[i,j] * m[i,j] + D2[i,j] * (1−m[i,j]) [2]
As shown by equation [2], since m takes on only values of 0 or 1, either the first half of the equation will have a value of zero, in which case the pixels from the second dataset are spliced into the image S, or the second half of the equation will have a value of zero, in which case the pixels from the first dataset will be spliced into the image S.” and [0061]-[0062], “If multiple organs of interest have been extracted, a mask for each organ may be provided. For example, if multiple organs have been extracted from a CT scan of a patient's chest, there will be a mask specifying the location of the lungs and a mask specifying a location of the ribs. ... In addition, there may be a default mask mn, which, for example, is intended to view all structures outside the extracted organ or organs ...”)
39. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 10 of Yan to include the disclosure of generating the second image as a difference image from the second mask image and the second fill image, of Novak. The motivation for this modification could have been to use the mask and fill images to focus on a region(s) or area(s) of interest and filter out any unnecessary noise or objects in the images. This will provide for clearer images, making it easier for a user to focus on a region while under examination.
40. Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Yan et al. (WO-2022/051977-A1, hereinafter "Yan") in view of Patriarche et al. (US-2019/0206036-A1, hereinafter "Patriarche").
41. As per claim 17, Yan discloses: The method of claim 8, wherein the first image and the second image are [[repeatedly]] registered with each other. (See rejection for claim 8.)
42. Yan doesn't explicitly disclose but Patriarche discloses: [[The method of claim 8, wherein the first image and the second image are]] repeatedly [[registered with each other.]] (Patriarche, [0013], “The flicker technique for detecting differences between images leverages the ability of the human visual system to detect motion ... Thus, to compare two images using the flicker technique, those images must be spatially registered, in order to ensure that the subregion that is actually changing, occupies the same region of the viewer's visual field in both images and to ensure that regions that aren't actually changing don't appear as abrupt intensity changes due simply to misalignment of the images being compared.” and [0034], “Some structures that might be desirable to compare using flicker have the capacity to spatially deform nonlinearly between acquisitions. Two examples include inflation or deflation of the lung and bending of the elbow. If two images of lungs that were differently inflated in the two images are rigidly registered and flickered, virtually every region of the image appears to be changing, due to the difference in inflation. However, detection of difference in inflation is not typically task-relevant. Thus, non-linear spatial registration (i.e. "warping") can be applied to the lungs, so that structures changing only due to normal biological function would be de-deformed, while structures changing due to an underlying disease process (e.g., the evolution of a tumor) are left alone. Thus, under flicker viewing mode, normal structures do not appear to change, while changing abnormal structures appear to change.” and [0035], “Another approach to flickering such deformable structures is to derive multiple rigid body spatial registrations, so that, for each location in image 1, there exists a rigid body registration transformation that is "most optimal" for aligning that local neighborhood to the corresponding local neighborhood in image 2.”)
43. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 8 of Yan to include the disclosure of
repeatedly registering the first image and the second image with each other, of Patriarche. The motivation for this modification could have been to ensure that during an examination process that any spatial and/or time changes that occur in the images continue to align the images with each other. This would help prevent image drift or changes where the images would become misaligned over time.
Conclusion
44. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. These are as follows: Craine et al. (US-6427022-B1), Fram et al. (US-2006/0093198-A1), and Rognin et al. (US-2008/0049994-A1).
45. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW CLOTHIER whose telephone number is (571)272-4667. The examiner can normally be reached Mon-Fri 8:00am-4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW CLOTHIER/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614