DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments and arguments filed on 3/31/2026 have been considered. Claim 1-19 are pending in the application of which claim 5 is withdrawn from consideration due to Applicant’s cancellation of the claim. Applicant’s amendments to the specifications have overcome each and every objection previously set forth in the Non-Final Office Action mailed on 1/30/2026.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 18, and 19 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Examiner acknowledges that in the 35 U.S.C 102 rejection in the previous Non-Final Office Action, Hirakawa (US 20200029789 A1) does not teach the amended portion of the claims which reads: “transforming the real image into areal-to-virtual transformed image having a virtual image style; and using the real-to-virtual transformed image as the real image and at least one of the first depth image or the second depth image in the viewpoint difference estimation processing.”.
In response: the combination of Tong (“Real-to-virtual domain transfer-based depth estimation for real-time 3D annotation in transnasal surgery: a study of annotation accuracy and stability") can be used to modify Hirakawa to fully teach the claims.
Tong teaches transform the real image into a real-to-virtual transformed image having a virtual image style (“Prior to depth estimation, a real-to-virtual image style transfer using cycle generative adversarial network (cycleGAN) [19] is performed.” - Introduction, Par 7, Lines 3-4); and use the real-to-virtual transformed image as the real image and at least one of the first depth image or the second depth image in the viewpoint difference estimation processing (“A difference ΔL between the distance L1 and the distance L2 which have been calculated in this manner represents a position deviation amount between the position of the endoscope distal end 3B and the position of a viewpoint at which a virtual endoscopic image K0 corresponding to the second depth image” – Hirakawa Par 65, Lines 1-5. [NOTE: After the combination, the generative adversarial network used to generate a real-to-virtual image as taught by Tong can modify Hirakawa to then convert the endoscope image to a virtual style image. Hirakawa can then perform their viewpoint difference estimation calculation between the virtual endoscopic image K0 which corresponds to a second depth image and the real-to-virtual image of the captured real endoscope image that was generated using the methods of Tong]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Hirakawa by incorporating the teachings of Tong to transform the real image to a real-to-virtual image and use the real-to-virtual image as the real image and at least one of the first or second depth images in the viewpoint difference estimation processing. One would be motivated to make this combination to provide the system with an image that best represents the real image without any noise (or at least reduced noised). Using the real-to-virtual image instead of the real image would improve the accuracy of the viewpoint difference estimation calculation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6, 9-14, and 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirakawa (US 20200029789 A1) and Tong et al (“Real-to-virtual domain transfer-based depth estimation for real-time 3D annotation in transnasal surgery: a study of annotation accuracy and stability”), hereinafter Hirakawa and Tong respectively.
Regarding claim 1, Hirakawa teaches an information processing apparatus comprising at least one processor (“An image acquisition unit”, Abstract), wherein the processor is configured to use at least one of a real image, which is captured by an endoscope inserted into a tubular structure of a subject and which represents an interior wall of the tubular structure (“the first medical image may be an actual endoscopic image which is generated using an endoscope inserted into a tubular structure of the subject and represents an inner wall of the tubular structure” – Par 12, Lines 2-5), or a first depth image, which represents a distance for each pixel from a viewpoint of the real image to the interior wall of the tubular structure (“the converted first medical image may be a first depth image representing a distance to the inner wall of the tubular structure from a viewpoint of the first medical image” – Par 12, Lines 9-12), and at least one of a virtual image, which is generated based on a three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a virtual viewpoint predetermined in the three-dimensional image (“the second medical image may be a virtual endoscopic image which is generated from a three-dimensional image including the tubular structure of the subject and spuriously represents the inner wall of the tubular structure” – Par 12, Lines 5-9), or a second depth image, which represents a distance for each pixel from the virtual viewpoint to the interior wall of the tubular structure (“the converted second medical image may be a second depth image representing a distance to the inner wall of the tubular structure from a viewpoint of the second medical image.” – Par 12, Lines 13-15) to perform viewpoint difference estimation processing of estimating a viewpoint difference between the viewpoint of the real image and the virtual viewpoint (“A difference ΔL between the distance L1 and the distance L2 which have been calculated in this manner represents a position deviation amount between the position of the endoscope distal end 3B and the position of a viewpoint at which a virtual endoscopic image K0 corresponding to the second depth image” – Par 65, Lines 1-5);
Hirakawa does not teach transform the real image into a real-to-virtual transformed image having a virtual image style; and use the real-to-virtual transformed image as the real image and at least one of the first depth image or the second depth image in the viewpoint difference estimation processing. However, Tong teaches transform the real image into a real-to-virtual transformed image having a virtual image style (“Prior to depth estimation, a real-to-virtual image style transfer using cycle generative adversarial network (cycleGAN) [19] is performed.” - Introduction, Par 7, Lines 3-4); and use the real-to-virtual transformed image as the real image and at least one of the first depth image or the second depth image in the viewpoint difference estimation processing (“A difference ΔL between the distance L1 and the distance L2 which have been calculated in this manner represents a position deviation amount between the position of the endoscope distal end 3B and the position of a viewpoint at which a virtual endoscopic image K0 corresponding to the second depth image” – Hirakawa Par 65, Lines 1-5. [NOTE: After the combination, the generative adversarial network used to generate a real-to-virtual image as taught by Tong can modify Hirakawa to then convert the endoscope image to a virtual style image. Hirakawa can then perform their viewpoint difference estimation calculation between the virtual endoscopic image K0 which corresponds to a second depth image and the real-to-virtual image of the captured real endoscope image that was generated using the methods of Tong]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Hirakawa by incorporating the teachings of Tong to transform the real image to a real-to-virtual image and use the real-to-virtual image as the real image and at least one of the first or second depth images in the viewpoint difference estimation processing. One would be motivated to make this combination to provide the system with an image that best represents the real image without any noise (or at least reduced noised). Using the real-to-virtual image instead of the real image would improve the accuracy of the viewpoint difference estimation calculation.
Regarding claim 18, the claim describes a method that performs the steps of claim 1. Therefore, method claim 18 corresponds to the apparatus disclosed in claim 1 and is rejected for the same reasons of anticipation as used above.
Regarding claim 19, the claim describes a non-transitory computer-readable storage medium (CRM) that performs the steps of apparatus claim 1. Therefore, CRM claim 19 corresponds to the apparatus disclosed in claim 1 and is rejected for the same reasons of anticipation as used above.
Regarding claim 2, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa further teaches wherein the processor is configured to at least use the first depth image and the second depth image to estimate the viewpoint difference in the viewpoint difference estimation processing (“the position estimation unit may estimate the position of the endoscope in the tubular structure based on the difference in pixel values between the first depth image and the second depth image.” – Par 14, Lines 2-5).
Regarding claim 6, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa does not teach wherein the processor is configured to use the real-to-virtual transformed image, the first depth image, the virtual image, and the second depth image to estimate the viewpoint difference in the viewpoint difference estimation processing. However, Tong further teaches wherein the processor is configured to use the real-to-virtual transformed image (“Prior to depth estimation, a real-to-virtual image style transfer using cycle generative adversarial network (cycleGAN) [19] is performed.” - Introduction, Par 7, Lines 3-4), and the first depth image (An adversarial network transfers the style from the real endoscopic view to a synthetic-like view for input into the depth estimation network, wherein framewise depth can be obtained in real time. – Abstract, Par 2. [NOTE: The depth estimation network converts the transformed real-to-virtual image into a depth map as show in in Fig.1].). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to further incorporate the teachings of Tong to utilize the depth image from the real-to-virtual transformation to estimate the viewpoint difference. When the real image is transformed to a virtual image, the real image has a more idealized lighting and sharper textures making for an easier comparison with the second virtual image. [NOTE: After the combination, the real-to-virtual image and first depth image from Tong’s disclosure can be used with Hirakawa’s teaching of the virtual image and second depth image to teach wherein the processor is configured to use the virtual image, and the second depth image to estimate the viewpoint difference in the viewpoint difference estimation processing.]
Regarding claim 9, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa further teaches wherein the first depth image is generated based on a pixel value of the real image (“the position estimation unit may estimate the position of the endoscope in the tubular structure based on the difference in pixel values between the first depth image and the second depth image” – Par 14, Lines 1-5.). [NOTE: It is widely known in the art that CT or MRI scans of the internal body provide depth images/maps. This map is created from the pixels of the image, based on the viewpoint of the endoscopic camera, which hold the distance between the camera and the organ wall.].
Regarding claim 10, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa does not teach wherein the first depth image is generated based on a pixel value of a real-to-virtual transformed image obtained by transforming the real image into a virtual image style. However, Tong further teaches, wherein the first depth image is generated based on a pixel value (“Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment”. [NOTE: depth images store a value to indicate how far each pixel in the image is from the object.]) of a real-to-virtual transformed image obtained by transforming the real image into a virtual image style (a real-to-virtual image style transfer using cycle generative adversarial network (cycleGAN), Introduction, Par 7, Lines 3-4).
PNG
media_image1.png
498
757
media_image1.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to further incorporate the teachings of Tong to base the first depth image on the pixel value of the transformed real-to-virtual image. It is common in the art to generate a depth image based on the pixel’s distance from an object. In this case, the depth images of Tong would be combined with Hirakawa viewpoint of the tubular structure to generate the depth image to find each pixel’s distance from the organ wall. Transforming the real image to the virtual image will allow the viewpoint estimation process to compare the virtual image with a real image that has idealized lighting and consistent textures when transformed to a virtual style image.
Regarding claim 11, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa further teaches wherein the second depth image is generated based on a pixel value of the virtual image (“the position estimation unit may estimate the position of the endoscope in the tubular structure based on the difference in pixel values between the first depth image and the second depth image” – Par 14, Lines 2-5). [NOTE: Hirakawa discloses that an estimation unit estimates the position of the endoscope based on pixels which implies that the depth images must have been created based on a pixel value. Generating the depth image for a virtual viewpoint would not require additional steps compared to a real image. In a similar fashion to the real endoscopic image, the pixels that the virtual viewpoint sees will determine how far each pixel is from the inner organ wall.].
Regarding claim 12, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa further teaches wherein the second depth image is generated based on distance information from the virtual viewpoint in the three-dimensional image to the interior wall of the tubular structure (“the converted second medical image may be a second depth image representing a distance to the inner wall of the tubular structure from a viewpoint of the second medical image.” – Par 12, Lines 13-15).
Regarding claim 13, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa does not teach wherein the processor is configured to use a depth image generation model that has been trained in advance to receive input of an image, which represents the interior wall of the tubular structure, and output a depth image, which represents a distance for each pixel from a viewpoint of the input image to the interior wall of the tubular structure, to generate at least one of the first depth image based on the real image or the second depth image based on the virtual image. However, Tong further teaches wherein the processor is configured to use a depth image generation model that has been trained in advance to receive input of an image (“As a result, a depth estimation network trained on synthetic endoscopic images can be deployed on real RGB endoscopic images” – Methodology, Par 3, Lines 3-4), which represents the interior wall of the tubular structure (“inside the virtual nasal airway model along a pre-defined pathway” – Methodology, Lines 2-3. [NOTE: tubular structure can refer to many internal organs such as the nasal airway.]), and output a depth image, which represents a distance for each pixel from a viewpoint of the input image to the interior wall of the tubular structure (“Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment”- Methodology, Par 7, Lines 3-5), to generate at least one of the first depth image based on the real image or the second depth image based on the virtual image (Tong provides Fig. 2 to show the predicted depth maps generated from synthetic-like images of a nasal airway model).
PNG
media_image2.png
167
450
media_image2.png
Greyscale
It would have been obvious to one of ordinary skill in the art to modify Hirakawa to incorporate the teachings of Tong to train a depth estimation network to generate a depth map to show the distance between the viewpoint and interior wall of the tubular structure. Training the depth estimation network using real and virtual images will generate accurate depth maps that give a close indication of how far the viewpoint is from the interior wall of the organ.
Regarding claim 14, Hirakawa in view of Tong teach the apparatus of claim 13. Hirakawa does not teach wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data including a combination of a virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a fourth viewpoint predetermined in the three-dimensional image; and a depth image for training, which is generated based on distance information from the fourth viewpoint in the three-dimensional image to the interior wall of the tubular structure and which represents a distance for each pixel from the fourth viewpoint to the interior wall of the tubular structure. However, Tong further teaches wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data (“we train a supervised depth estimation network in a virtual environment and utilize it to predict depth of real endoscopic image” – Introduction, Par 7) including a combination of: a virtual image for training , which is generated based on the three-dimensional image of the subject (“A dataset of 3600 synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2. [NOTE: synthetic endoscopic images referring to virtual 3D image of the interior of tubular structure]) and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a fourth viewpoint predetermined in the three-dimensional image (“inside the virtual nasal airway model along a pre-defined pathway” – Methodology, Lines 2-3 [NOTE: the “fourth viewpoint” is interpreted as another instance to differentiate between the viewpoints from other embodiments of the present invention.]); and a depth image for training (“A dataset of 3600 synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2. [NOTE: ground truth depth maps are the depth images used for training the depth image generation model]), which is generated based on distance information from the fourth viewpoint in the three-dimensional image to the interior wall of the tubular structure (“synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2. [NOTE: the virtual endoscopic images were paired with their respective ground truth depth maps which hold each pixel’s distance from the virtual viewpoint to the interior wall of the tubular structure.]) and which represents a distance for each pixel from the fourth viewpoint to the interior wall of the tubular structure (“Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment”- Methodology, Par 7, Lines 3-5. [NOTE: the depth value could be a representation of the distance between the viewpoint and inner organ wall.]). It would have been obvious to one of ordinary skill in the art to modify the teachings of Hirakawa to incorporate the teachings of Tong to use their depth estimation network to generate depth maps after training it with a virtual image based on the 3D image of the tubular structure and a depth image based on the pixel’s distance from the tubular structure inner wall. This training approach ensures that the depth estimation model properly develops an accurate depth map with respect to the viewpoints.
Claim(s) 3, 4, and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirakawa, Tong, and Lopez et al (US 11398048 B2), hereinafter Lopez.
Regarding claim 3, Hirakawa in view of Tong teaches the apparatus of claim 1. Hirakawa further teaches wherein the processor is configured to: generate at least one of the first depth image (“the converted first medical image may be a first depth image”, Par 12) or the second depth image (“the converted second medical image may be a second depth image”, Par 12). Hirakawa does not teach using a viewpoint difference estimation model that has been trained in advance to at least receive input of at least one of the first depth image or the second depth image and output the viewpoint difference by using the input of at least one of the first depth image or the second depth image, in the viewpoint difference estimation processing. However, Lopez teaches using a viewpoint difference estimation model (“The inertial measurement unit pose prediction, and the neural network pose prediction are combined in order to estimate the current camera pose” – Abstract. [NOTE: Lopez uses a neural network model to estimate the camera pose of an image at one point in time and another image some time before the current image given depth maps as input.]) that has been trained in advance to at least receive input of at least one of the first depth image or the second depth image (“The depth maps generated by the first neural network NN1 are generated sequentially for each image frame, and then input to the second neural network NN2” – Col 9, Lines 27-30. [NOTE: NN1 generates the depth maps which are then fed as input to NN2 which estimates the camera pose.]) and output the viewpoint difference by using the input of at least one of the first depth image or the second depth image, in the viewpoint difference estimation processing (“FIG. 4 illustrates in more detail a schematic diagram of an example neural network NN that includes a first neural network NN1 that predicts a depth map, and a second neural network NN2 that estimates a pose.” - Col 9, Lines 33-36 [NOTE: NN2, as disclosed by Lopez, estimates the camera pose which is the viewpoint/orientation of the camera given the depth maps generated from NN1]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teachings of Lopez to train a machine learning model to estimate the camera pose using a depth map of an image as input. One of ordinary skill in the art could also apply the same model to a second image and therefore can find the viewpoint difference when comparing the results of both estimations. This approach allows the user to see how far off the images are from each other which allows them to adjust the camera viewpoint to align the orientation of both images.
Regarding claim 4, Hirakawa in view of Tong and Lopez teaches the apparatus of claim 3. Hirakawa further teaches wherein the processor is configured to: generate at least one of the first depth image (“the converted first medical image may be a first depth image”, Par 12) or the second depth image (“the converted second medical image may be a second depth image”, Par 12). Hirakawa does not teach using a viewpoint difference estimation model that has been trained in advance to at least receive input of the first depth image and the second depth image and output the viewpoint difference by using the input first depth image and second depth image, in the viewpoint difference estimation processing. However, Lopez teaches using a viewpoint difference estimation model that has been trained in advance to at least receive input of the first depth image and the second depth image (“The system performs operations, including: generating, using one or more neural networks, a neural network pose prediction for the current image frame; and adjusting a previous camera pose using inertial measurement unit data representing a motion of the camera between the previous point in time and the current point in time, to provide an inertial measurement unit pose prediction for the current point in time” – Abstract. [NOTE: two separate image frames are taken to find the difference in camera pose between both frames. The images were taken a set time apart from each other. Depth maps are generated for each image frame resulting in two image frames being compared].) and output the viewpoint difference by using the input first depth image and second depth image, in the viewpoint difference estimation processing (“FIG. 4 illustrates in more detail a schematic diagram of an example neural network NN that includes a first neural network NN1 that predicts a depth map, and a second neural network NN2 that estimates a pose.” - Col 9, Lines 33-36 [NOTE: the viewpoint difference estimation could then be obtained from NN2 which generates the predicted camera pose between the first current image and second prior image.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teachings of Lopez to have both the first and second depth map as input for the viewpoint estimation model. Doing so would allow the estimation model to analyze both depth maps simultaneously in order to generate an accurate estimation.
Regarding claim 7, Hirakawa in view of Tong and Lopez teaches the apparatus of claim 3. Hirakawa does not teach wherein the viewpoint difference estimation model is a model that has been trained through supervised learning using training data including a combination of: at least one of a first virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a first viewpoint predetermined in the three-dimensional image, or a first depth image for training, which represents a distance for each pixel from the first viewpoint to the interior wall of the tubular structure; at least one of a second virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a second viewpoint that is predetermined in the three-dimensional image and that is different from the first viewpoint, or a second depth image for training, which represents a distance for each pixel from the second viewpoint to the interior wall of the tubular structure; and a viewpoint difference between the first viewpoint and the second viewpoint. However, Lopez teaches wherein the viewpoint difference estimation model is a model that has been trained through supervised learning using training data (“When supervised training is used to train the one or more neural networks NN to generate a pose, the training involves adjusting the parameters of the neural network such that for each training image frame, a loss function based on a difference between the neural network pose prediction P.sup.NN, and the training image frame's corresponding previously-labelled camera pose data, meets a stopping criterion.” – Col 10, Lines 28-54) including a combination of: at least one of a first depth image for training, a second depth image for training (“Supervised training involves setting the parameters of the neural network using training image frames that are previously-labelled with corresponding camera pose or depth map data.” – Col 10, Lines 11-14. [NOTE: depth map data refers to the depth images that correspond to the training image frames]), and a viewpoint difference between the first viewpoint and the second viewpoint (“RNNs are suited to determining differences, and may therefore be used to determine a change in pose between two image frames.” – Col 9 , Lines 54-56 [NOTE: each image frame will have a viewpoint which is compared together to estimate the viewpoint difference]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teachings of Lopez to include a first and second depth image for training of the viewpoint difference estimation model. Having the model look at both depth images simultaneously will help the model develop the correct recognition patterns to accurately estimate the difference in viewpoint/orientation.
Hirakawa in view of Lopez still does not teach training data including a combination of: at least one of a first virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a first viewpoint predetermined in the three-dimensional image, or a first depth image for training, which represents a distance for each pixel from the first viewpoint to the interior wall of the tubular structure; at least one of a second virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a second viewpoint that is predetermined in the three-dimensional image and that is different from the first viewpoint, or a second depth image for training, which represents a distance for each pixel from the second viewpoint to the interior wall of the tubular structure. However, Tong teaches training data including a combination of: at least one of a first virtual image for training (“A dataset of 3600 synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2), which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a first viewpoint predetermined in the three-dimensional image (“inside the virtual nasal airway model along a pre-defined pathway” – Methodology, Lines 2-3), or a first depth image for training, which represents a distance for each pixel from the first viewpoint to the interior wall of the tubular structure (“Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment”- Methodology, Par 7, Lines 3-5); at least one of a second virtual image for training (“A dataset of 3600 synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2. [NOTE: It would be obvious that if a first virtual image is used for training, then a second one can be provided for training.]), which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a second viewpoint that is predetermined in the three-dimensional image and that is different from the first viewpoint (“inside the virtual nasal airway model along a pre-defined pathway” – Methodology, Lines 2-3. [NOTE: it is common for a virtual model, such as the virtual nasal airway model, to have a predetermined viewpoint in order for the user to define its position in the virtual model. This can be combined with Lopez’s teaching of comparing the viewpoints of an image taken in a current point in time in a previous point in time by substituting one of these images with the virtual viewpoint image.]), or a second depth image for training, which represents a distance for each pixel from the second viewpoint to the interior wall of the tubular structure (“Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment”- Methodology, Par 7, Lines 3-5). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa in view of Lopez to further incorporate the teachings of Tong to include a combination of a first virtual image (or first depth map) and a second virtual image (or second depth map) for training of the virtual viewpoint difference estimation model. The virtual images will offer a predetermined viewpoint that is known and defined which makes it easier for the difference estimation model to recognize patterns and generate an estimated difference.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirakawa, Tong, Lopez, and Shen (“Context-Aware Depth and Pose Estimation for Bronchoscopic Navigation”), hereinafter Shen.
Regarding claim 8, Hirakawa in view of Tong and Lopez teaches the apparatus of claim 3. Hirakawa does not teach wherein the viewpoint difference estimation model is a model that has been trained through supervised learning using training data including a combination of: at least one of a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure, or a depth image for training, which represents a distance for each pixel from a viewpoint of the real image for training to the interior wall of the tubular structure; at least one of a virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a third viewpoint predetermined in the three-dimensional image, or a virtual depth image for training, which represents a distance for each pixel from the third viewpoint to the interior wall of the tubular structure; and a viewpoint difference between the viewpoint of the real image for training and the third viewpoint. However, Lopez teaches wherein the viewpoint difference estimation model is a model that has been trained through supervised learning using training data (“When supervised training is used to train the one or more neural networks NN to generate a pose, the training involves adjusting the parameters of the neural network such that for each training image frame, a loss function based on a difference between the neural network pose prediction P.sup.NN, and the training image frame's corresponding previously-labelled camera pose data, meets a stopping criterion.” – Col 10, Lines 28-54) including a combination of: at least one of a first depth image for training, a second depth image for training (“Supervised training involves setting the parameters of the neural network using training image frames that are previously-labelled with corresponding camera pose or depth map data.” – Col 10, Lines 11-14), and a viewpoint difference between the first viewpoint and the second viewpoint (“RNNs are suited to determining differences, and may therefore be used to determine a change in pose between two image frames.” – Col 9 , Lines 54-56 [NOTE: each image frame will have a viewpoint which is compared together to estimate the viewpoint difference]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teachings of Lopez to include a first and second depth image for training of the viewpoint difference estimation model. Having the model look at both depth images simultaneously will help the model develop estimate the differences between each image more accurately.
Hirakawa in view of Lopez still does not teach training data including a combination of: at least one of a first virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a first viewpoint predetermined in the three-dimensional image, or a first depth image for training, which represents a distance for each pixel from the first viewpoint to the interior wall of the tubular structure; at least one of a second virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a second viewpoint that is predetermined in the three-dimensional image and that is different from the first viewpoint, or a second depth image for training, which represents a distance for each pixel from the second viewpoint to the interior wall of the tubular structure. However, Tong teaches training data including a combination of: at least one of a virtual image for training (“A dataset of 3600 synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2), which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a third viewpoint predetermined in the three-dimensional image (“inside the virtual nasal airway model along a pre-defined pathway” – Methodology, Lines 2-3. [NOTE: the “third viewpoint” is interpreted as another instance to differentiate between the viewpoints from other embodiments of the present invention.]), or a virtual depth image for training, which represents a distance for each pixel from the third viewpoint to the interior wall of the tubular structure (“Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment”- Methodology, Par 7, Lines 3-5. [NOTE: a virtual depth image must have been created if a synthetic model of the inner organ was used.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa in view of Lopez to further incorporate the teachings of Tong to include a virtual image (or depth map of the virtual image) as training data for the viewpoint difference estimation model. Although Tong uses this training data for a depth estimation network, it can be combined by one of ordinary skill in the art to include the virtual images as training for the viewpoint difference estimation network instead. Doing so would allow the network to recognize patterns that can accurately estimate the camera pose of each image in order to find the difference.
Hirakawa in view of Lopez and Tong still does not teach including at least one of a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure, or a depth image for training, which represents a distance for each pixel from a viewpoint of the real image for training to the interior wall of the tubular structure. However, Shen teaches including at least one of a real image for training , which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure (“the paired CT scans and bronchoscopic videos from two subjects for training and data from another subject for testing” – Experiments , Par 1, Lines 10-12), or a depth image for training, which represents a distance for each pixel from a viewpoint of the real image for training to the interior wall of the tubular structure (Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment”- Methodology, Par 7, Lines 3-5. [NOTE: the depth image can also easily be generated by one of ordinary skill in the art for a real image when given the endoscopic images of the inner wall of the tubular structure.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teaching of Shen to include a real image (or depth map based on the real image) to be used as training for the viewpoint difference estimation model. Although not explicitly mentioned, one of ordinary skill in the art could use the depth maps intended for training the depth estimation model in Tong for the viewpoint difference estimation model in Lopez. Doing so would give the viewpoint difference estimation model better accuracy when analyzing images or depth maps that are not ideal compared to a virtual idealistic image.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirakawa, Tong and Oda et al (Realistic endoscopic image generation method using virtual-to-real image-domain translation), hereinafter Oda.
Regarding claim 15, Hirakawa in view of Tong teaches the apparatus of claim 13. Hirakawa does not teach wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data including a combination of: an image obtained by transforming a virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a fifth viewpoint predetermined in the three-dimensional image, into a real image style by using a transformation model that has been trained in advance to receive input of the virtual image, transform the input virtual image into the real image style, and output the transformed virtual image; and a depth image for training, which is generated based on distance information from the fifth viewpoint in the three-dimensional image to the interior wall of the tubular structure and which represents a distance for each pixel from the fifth viewpoint to the interior wall of the tubular structure. However, Tong further teaches wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data (“we train a supervised depth estimation network in a virtual environment and utilize it to predict depth of real endoscopic image” – Introduction, Par 7) including a depth image for training, which is generated based on distance information from the fifth viewpoint in the three-dimensional image to the interior wall of the tubular structure ( “synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2 [NOTE: depth maps for training contain distance information from the viewpoint to the interior organ wall. Also, the “fifth viewpoint” is interpreted as another instance to differentiate between the viewpoints from other embodiments of the present invention.]) and which represents a distance for each pixel from the fifth viewpoint to the interior wall of the tubular structure (“Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment” - Methodology Par 7, Lines 3-5). It would have been obvious to one of ordinary skill in the art to modify the teachings of Hirakawa to incorporate the teachings of Tong have the depth estimation model use depth maps for training based on the distance information from the viewpoint to the inner tubular wall. By using depth image for training, the model becomes more accurate in predicting the distance from the inner organ wall based on the viewpoint.
Hirakawa in view of Tong still does not teach including an image obtained by transforming a virtual image for training, which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a fifth viewpoint predetermined in the three-dimensional image, into a real image style by using a transformation model that has been trained in advance to receive input of the virtual image, transform the input virtual image into the real image style, and output the transformed virtual image. However, Oda teaches including an image obtained by transforming a virtual image for training (“For the training, virtual endoscopic images
PNG
media_image3.png
18
144
media_image3.png
Greyscale
and real endoscopic images
PNG
media_image4.png
21
141
media_image4.png
Greyscale
are used” – Section 2.3, Par 3), which is generated based on the three-dimensional image of the subject and which represents, in a pseudo manner, the interior wall of the tubular structure as viewed from a fifth viewpoint predetermined in the three-dimensional image (“We generate virtual endoscopic images from CT volumes” – Section 2.2, Par 1, Fig 1. [NOTE: Fig.1 shows a set of 3D virtual endoscopic images based on a human colon]), into a real image style by using a transformation model that has been trained in advance to receive input of the virtual image (“They improve the reality of the virtual endoscopic images using a virtual-to-real image-domain translation technique. The image-domain translator is implemented as a fully convolutional network (FCN).” - Abstract), transform the input virtual image into the real image style, and output the transformed virtual image (“improve the reality of the virtual endoscopic images using a virtual-to-real image-domain translation technique” - Abstract). It would have been obvious to one of ordinary skill in the art to modify the teachings of Hirakawa to incorporate the teachings of Oda to include the transforming of a virtual image to a real image style for training of the depth estimation network. This approach gains the advantage of analyzing a virtual image that includes textures and lighting that would be expected of a real endoscopic image. This will allow the depth estimation model to generate depth maps that would closely fit real endoscopic images.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirakawa, Tong and Shen.
Regarding claim 16, Hirakawa in view of Tong teaches the apparatus of claim 13. Hirakawa does not teach wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data including a combination of: a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure; and a depth image for training, which is generated based on distance information from a viewpoint corresponding to a viewpoint at which the real image for training is captured, in the three-dimensional image of the subject, to the interior wall of the tubular structure and which represents a distance for each pixel from the viewpoint corresponding to the viewpoint at which the real image for training is captured to the interior wall of the tubular structure. However, Tong teaches wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data including a depth image for training (“we train a supervised depth estimation network in a virtual environment and utilize it to predict depth of real endoscopic image” – Introduction, Par 7), which is generated based on distance information from a viewpoint corresponding to a viewpoint at which the real image for training is captured, in the three-dimensional image of the subject, to the interior wall of the tubular structure (synthetic endoscopic images and the corresponding ground truth depth maps were captured” – Methodology, Par 2, Lines 1-2. [NOTE: depth maps for training contain distance information from the viewpoint to the interior organ wall.]) and which represents a distance for each pixel from the viewpoint corresponding to the viewpoint at which the real image for training is captured to the interior wall of the tubular structure (Depth at each pixel was stored as a normalized float number in a range between 0 (far) and 1 (near), which was then converted into a depth range of 0.01–25 mm in the virtual environment” – Methodology Par 7, Lines 3-5). It would have been obvious to one of ordinary skill in the art to modify the teachings of Hirakawa to incorporate the teachings of Tong to train a depth estimation model with a depth image based on the viewpoint’s distance from the inner wall of the tubular structure. The depth image provides insight on how far each pixel of the viewpoint is from the inner wall of the organ. Using these depth maps to train the depth estimation network makes the generation of depth maps more accurate.
Hirakawa in view of Tong still does not teach including a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure. However, Shen teaches including a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure (“the paired CT scans and bronchoscopic videos from two subjects for training and data from another subject for testing” – Experiments , Par 1, Lines 10-12. [NOTE: the real images are collected from the CT scans and videos from the endoscope.]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teachings of Shen to include real images from the endoscope for training of the depth estimation network taught by Tong. The real images provide a realistic s of information that will train the depth estimation to look at non-ideal images which will strengthen the accuracy when generating depth maps.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirakawa, Tong, Shen, and Yashiro et al (US 20180092515 A1), hereinafter Yashiro.
Regarding claim 17, Hirakawa in view of Tong teaches the apparatus of claim 13. Hirakawa does not teach wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data including a combination of: a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure; and a depth image for training, which is generated based on an actual measurement value of a distance from a viewpoint at which the real image for training is captured to the interior wall of the tubular structure, the actual measurement value being obtained by a distance-measuring sensor mounted on the endoscope, and which represents a distance for each pixel from the viewpoint at which the real image for training is captured to the interior wall of the tubular structure. However, Tong further teaches wherein the depth image generation model is a model that has been trained in advance through supervised learning using training data (“we train a supervised depth estimation network in a virtual environment and utilize it to predict depth of real endoscopic image” – Introduction, Par 7). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate Tong to train a depth estimation network through supervised learning. This approach to training a machine learning model allows the model to recognize patterns and make accurate predictions when given the real or virtual endoscopic images. This would yield predictable results since supervised learning is very common in the art.
Hirakawa in view of Tong still does not teach a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure; and a depth image for training, which is generated based on an actual measurement value of a distance from a viewpoint at which the real image for training is captured to the interior wall of the tubular structure, the actual measurement value being obtained by a distance-measuring sensor mounted on the endoscope, and which represents a distance for each pixel from the viewpoint at which the real image for training is captured to the interior wall of the tubular structure. However, Shen teaches including a real image for training, which is captured by the endoscope inserted into the tubular structure of the subject and which represents the interior wall of the tubular structure (“the paired CT scans and bronchoscopic videos from two subjects for training and data from another subject for testing” – Experiments , Par 1, Lines 10-12). It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teachings of Shen to include real images as part of the training for the depth estimation network taught by Tong. Using real images trains the model to recognize scenarios that are non-ideal as compared to virtual images. This will strengthen the model’s prediction accuracy.
Hirakawa in view of Tong and Shen still does not teach a depth image for training, which is generated based on an actual measurement value of a distance from a viewpoint at which the real image for training is captured to the interior wall of the tubular structure, the actual measurement value being obtained by a distance-measuring sensor mounted on the endoscope. However, Yashiro teaches a depth image for training, which is generated based on an actual measurement value of a distance from a viewpoint at which the real image for training is captured to the interior wall of the tubular structure, the actual measurement value being obtained by a distance-measuring sensor mounted on the endoscope (“As shown in FIG. 43, the distance measurement unit 706 measures a distance D1 between the tip part 12d and the object to be observed near boundaries between the field 501 of view of the direct-viewing observation unit 41” – Par 179, Lines 20-27). Although not explicitly mentioned, one of ordinary skill in the art could use the process to obtain measurements from Yashiro to generate the depth images as disclosed in Tong in order to be used as training data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to modify Hirakawa to incorporate the teachings of Yashiro to use the sensor mounted on the endoscope to obtain the measurements between the viewpoint and inner wall. The distance information can be used to generate the depth image as taught by Tong to then be used for training the depth estimation network to increase the accuracy of predicting and generating depth maps. [NOTE: After the combination, the process of obtaining the actual measurement between the viewpoint and the inner wall using a sensor as taught by Yashiro can then be applied to Tong’s depth images used to train the depth estimation network which teaches a depth image for training which represents a distance for each pixel from the viewpoint at which the real image for training is captured to the interior wall of the tubular structure].
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID V. NGUYEN whose telephone number is (571)272-6111. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID VAN NGUYEN/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights