DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
2. This communication is in response to the application filled on 08/28/2024.
3. Claims 1-19 are pending.
4. Limitations appearing inside {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Information Disclosure Statement
5. The information disclosure statement (IDS) submitted on 10/08/2024 has been considered by the examiner.
Claim Rejections - 35 USC § 103
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
7. Claims 1-5, 7, 11-13, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over “Hands-Free System for Bronchoscopy Planning and Guidance” to Khare et al. (hereinafter Khare) and further in view of U.S. Patent No. 7,951,070 to Ozaki et al. (hereinafter Ozaki).
8. Regarding Claim 1, Khare discloses an image processing apparatus comprising:
a processor,
wherein the processor displays, on a display device, a moving image that is generated based on volume data including a bronchial image showing a bronchus and that shows an aspect in which an inside of the bronchus is observed ([pg. 2795, col. 2, II. Methods, par. 1, ln. 1-6] “Our VBN system consists of a guidance computer, monitor, keyboard, and mouse, all mounted on a mobile cart. The computer is a mid-range Dell T5400 64-bit workstation with a 3-GHz quad-core Xeon processor (16-GB RAM, 768M NVidia graphics card). The computer draws upon a Matrox Solios eA frame grabber for video stream interfacing.”, [pg. 2796, col. 1, 2) Image-Guided Bronchoscopy, par. 1, ln. 1-14] “Image-Guided Bronchoscopy Performed in the bronchoscopy suite after patient preparation, two steps are involved. a) System initialization: Interface the guidance computer to the bronchoscopy hardware and preview the navigation plan. b) Live bronchoscopy: The physician navigates the bronchoscope toward each ROI using guidance in formation supplied by the system’s monitor and pre computed navigation plan. During navigation, the physician uses the foot switch to invoke commands that update the guidance information. If desired, the physician can also invoke a position-verification mechanism to confirm and, if indicated, correct the bronchoscope’s position.”, [pg. 2796, col. 2, A. Conceptual Elements, par. 1, ln. 18] “For this paper, we follow and enhance when necessary some of the notational conventions of Gibbs et al. [16]. To begin, let the patient’s high-resolution 3-D chest CT scan define the virtual space, while the patient’s actual chest anatomy defines the real space. When navigating through the airways of the CT-based virtual space, we use a virtual camera, or virtual bronchoscope, to image the virtual airway tree’s interior [13], [15]. Similarly, the “real” bronchoscope camera images the patient’s airway tree interior during the procedure. Let
I
C
T
Θ
denote the virtual bronchoscope’s view rendered at pose 𝝝 within virtual space, where the virtual camera’s pose 𝝝 is given by the standard six parameter vector defining the pose’s 3-D spatial position and Euler angles. View
I
C
T
Θ
, which is a CT-derived edoluminal rendering of the airway-tree interior, will be referred to as a VB view. Analogously, let
I
V
Θ
represent the bronchoscope camera’s view at pose 𝝝 within the patient’s airway tree of real space. Each 2-D video frame of the bronchoscope’s live video stream gives such a view.”, [pg. 2798, Fig. 2] see virtual bronchus image of inside of the bronchus, [pg. 2800, Fig. 4] see both virtual and live video stream of interior of bronchus),
receives an instruction ([pg. 2800, col. 1, C. Image-Guided Bronchoscopy, par. 1, ln. 1 to col. 2, par. 1 ln. 12] “Upon completing procedure planning, the VBN system can be wheeled into the bronchoscopy suite. After patient prepping, image-guided bronchoscopy can then proceed. This involves 1) system initialization followed by 2) live bronchoscopy. To perform system initialization, we first interface the system to the bronchoscope’s output video source and invoke the system software, including the software driving the foot switch. Next, the patient’s CT scan, previously computed CT-based 3-D chest model, and navigation plan are loaded. The foot switch, positioned on the floor during bronchoscopy, has its three switches programmed with the following functions: 1) “rotate/advance,” to invoke rotation and advance commands; 2) “verify,” to invoke location verification; and 3) “back,” to enable backing up to a previous airway branch. Note that the foot switch is the preferred method for system operation. However, we also provide analogous keyboard and mouse commands that can override the foot switch. In addition, the bronchoscope’s video monitor is always available. These redundancies provide important fail safes in case of system failures, software crashes, or unexpected adverse events.”) {performed by a voice},
acquires a positional relationship between a plurality of positions from an upstream side to a downstream side in the bronchus according to the instruction ([pg. 2796, col. 2, A. Conceptual Elements, par. 2, ln. 1 to pg. 2797, col. 1, par. 2, ln. 12] “As is done in all IGI bronchoscopy systems, the airway centerlines serve as the natural pathways for navigating the airway tree [12], [16], [21], [35], [36]. We represent the centerlines by a discrete set of airway branches
b
j
, where
b
j
consists of an ordered discrete set of
w
j
view sites; i.e.
b
j
=
{
v
j
,
1
,
v
j
,
2
,
…
v
j
,
w
j
}
(1) where
v
j
,
1
is the ith view site constituting
b
j
. For our work, a view site is specified by its 3-D location within virtual space, along with vectors defining the field of view and parameters measuring the local branch angle and branch diameter [16]. Finally, each branch
b
j
has an associated branch length
l
j
, defined as the Euclidean distance between the first and last view sites of
b
j
. For each ROI, let r denote a route that defines a pathway through the airway tree leading to the ROI. Route r consists of an ordered set of branches, decomposed into view sites per (1), leading from the trachea to a final destination near the ROI, i.e.,
r
=
b
1
,
b
2
,
…
,
b
j
,
…
,
b
a
,
b
a
+
1
2
=
v
1
,
…
,
v
1
,
w
1
,
…
,
v
2
,
w
2
,
v
3,2
,
…
,
v
3
,
w
3
,
…
v
a
,
w
a
,
v
a
+
1
,
2
,
…
v
D
(3) where
v
D
denotes the destination view site. All routes begin with the airway tree’s unique tracheal branch
b
1
. Regarding (2), route r by conversion consists of a complete branches and a partial (a+1)th branch. Per (1), the final view site of branch
b
j
, namely
v
j
,
w
j
, also equals the first view site
v
j
+
1
,
1
of
b
j
+
1
. Note, of course, that a can differ from different routes. Equation (3) will be useful, because it highlights route r’s a endpoint
v
j
,
w
j
=
v
j
+
1
,
1
,
j
=
1
,
2
,
…
,
a
.
(4) As discussed in Section II-B, the method for navigation plan computation (Algorithm 1) uses view sites near the locations (4) to observe successive branch bifurcations along the route. Given the definitions above, the goal of image-guided bronchoscopy is to assist the physician in navigating the “real” bronchoscope through the patient’s airway tree so that he/she follows the preplanned airway route r defined in virtual space. This suggests that, as the virtual bronchoscope moves along route r , the real bronchoscope should appear to be at the same location. In other words, during the live procedure, the bronchoscope’s video stream should appear to stay approximately synchronized, or registered, with the virtual bronchoscope’s VB view sequence along r. As Section II-C will make clear, our system’s guidance strategy (Algorithm 2) expects this synchronization to occur near r’s bifurcation points (4), per the relation
I
V
Θ
j
'
≈
I
C
T
Θ
j
,
j
=
1,2
,
…
,
a
(5) where
I
V
Θ
j
'
is a frame in the video stream,
I
C
T
Θ
j
is a VB view, and
Θ
j
≜
virtual-space pose near branch
b
j
’s endpoint
v
j
,
w
j
on route r
Θ
j
'
≜
real-space pose approximately corresponding to the same viewpoint a
Θ
j
. The physician, at times, may also use the system’s position verification mechanism to further validate bronchoscope location.”, [pg. 2798, Fig. 2] see virtual bronchus image of inside of the bronchus which correspond specifically to a bifurcation, [pg. 2800, Fig. 4] see both virtual and live video stream of interior of bronchus that correspond to multiple upstream and downstream bifurcations along a route r), and
displays the moving image in a display aspect corresponding to the positional relationship ([pg. 2796, col. 1, 2) Image-Guided Bronchoscopy, par. 1, ln. 1-14], [pg. 2796, col. 2, A. Conceptual Elements, par. 1, ln. 18], [pg. 2796, col. 2, A. Conceptual Elements, par. 2, ln. 1 to pg. 2797, col. 1, par. 2, ln. 12], [pg. 2798, Fig. 2] see virtual bronchus image of inside of the bronchus, [pg. 2800, Fig. 4] see both virtual and live video stream of interior of bronchus, see also lines corresponding to the pathway r).
Khare does not specifically disclose wherein the instruction is performed by voice.
However, Ozaki specifically teaches wherein the apparatus receives an instruction performed by voice for a bronchoscopy operation ([col. 8, ln. 54-63] “The system controller 17 has a microphone 24 for capturing voice as instructing means. The microphone 24 is removably connected to the system controller 17 through a signal cable extending from the head set 25. The microphone 24 may be a pin microphone. The system controller 17 and the head set 25 may be adjusted to communicate voice information by radio communication such as infrared rays. The microphone 24 may be attached to a goggles type or glasses type apparatus called face mount display (FMD) or head mount display (HMD).”, [col. 14, ln. 26-33] “With an endoscopic surgery system 1A of this embodiment, an operator can give an instruction by voice directly through a microphone 24 and easily operate. Thus, a desired rendering image of a surrounding of a target part can be obtained. An operator may give instructions by using not only the microphone 24 but also a mouse and/or keyboard or a remote controller.”, [col. 16, ln. 43 to col. 17, ln. 32] “Here, an operator instructs to "synthesize images" through the microphone 24 with respect to a body-cavity rendering image (refer to FIG. 12) of a target part and surroundings displayed on the rendering monitor 27. Thus, the rendering image creating apparatus 28A judges whether a voice instruction from the microphone 24 is given or not (step S32). If the voice instruction is "synthesize images", the three processing pattern images are read out from the pattern image storing section 33 and the three processing pattern images are synthesized as shown in FIG. 13 (step S33). The resulting synthesized image is displayed on the rendering monitor 27. Then, in accordance with progress of the surgery, the operator gives a voice instruction from the microphone 24 and displays a desired rendering image on the rendering monitor 27 with respect to the synthesized image… The rendering image creating apparatus 28A may perform subtraction processing as shown in FIG. 14 as synthesizing processing. More specifically, in response to a voice instruction, "part before target organ, delete" by the operator, the rendering image creating apparatus 28A subtracts an image of a part before the target organ from the synthesized image and displays an image having blood vessels in the target organ on the rendering monitor 27… The endoscopic surgery system 1A may obtain a desired rendering image with respect to a synthesized image by directly instructing selective display as shown in FIG. 15 regardless of progress of the surgery. More specifically, in response to a voice instruction, "target organ image" by an operator, the rendering image creating apparatus 28A reads out "target organ image" data from processing pattern images stored in the pattern image storing section 33 and directly switches from a synthesized image to the target organ image.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Khare and Ozaki as within the same field of hands-free endoscopic imaging systems, and as analogous to the claimed invention. The motivation to combine would have been obvious to one of ordinary skill in the art, in that the voice command of Ozaki is an alternative to foot pedal as described in Khare, and would offer an analogous hands-free system if implemented in place of the pedal as described in Khare. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the apparatus of Khare with the instruction performed by voice of Ozaki through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 1.
9. Regarding Claim 2, a combination of Khare and Ozaki teaches the apparatus of claim 1. Khare discloses wherein the plurality of positions include a position corresponding to an upstream bifurcation in the bronchus and a position corresponding to a downstream bifurcation in the bronchus ([pg. 2796, col. 1, 2) Image-Guided Bronchoscopy, par. 1, ln. 1-14], [pg. 2796, col. 2, A. Conceptual Elements, par. 1, ln. 18], [pg. 2796, A. Conceptual Elements, par. 2, ln. 1 to pg. 2797, col. 1, par. 2, ln. 12], [pg. 2798, Fig. 2] [pg. 2800, Fig. 4]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 2.
10. Regarding Claim 3, a combination of Khare and Ozaki teaches the apparatus of claim 2. Khare discloses wherein the position corresponding to the upstream bifurcation is a position upstream of the upstream bifurcation in the bronchus, and the position corresponding to the downstream bifurcation is a position upstream of the downstream bifurcation in the bronchus ([pg. 2796, col. 1, 2) Image-Guided Bronchoscopy, par. 1, ln. 1-14], [pg. 2796, col. 2, A. Conceptual Elements, par. 1, ln. 18], [pg. 2796, A. Conceptual Elements, par. 2, ln. 1 to pg. 2797, col. 1, par. 2, ln. 12], [pg. 2798, Fig. 2] [pg. 2800, Fig. 4] see (a) and then (c)). The examiner notes this further would have been obvious to one of ordinary skill in the art in that the bifurcations of the lung form a tree structure, and thus for every bifurcation, there is only one path to that bifurcation from the starting point (i.e. the trachea). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 3.
11. Regarding Claim 4, a combination of Khare and Ozaki teaches the apparatus of claim 3. Khare further discloses wherein the display aspect includes an aspect in which the display of the moving image is stopped at the position corresponding to the upstream bifurcation and at the position corresponding to the downstream bifurcation, an aspect in which a speed at which the display of the moving image is advanced is decreased, or an aspect in which the display of the moving image is stopped after the speed at which the display of the moving image is advanced is decreased ([pg. 2802, col. 2, par. 2, ln. 1 to pg. 2803, col. 1, par. 2, ln. 8] “The physician then invokes the “rotate” foot-switch command. This action causes the guidance display to highlight the CCW-rotated VB view at the main carina
I
C
T
Θ
1
T
in the navigation plan tool’s advance column for row 1 [see Fig. 4(b)]. It also results in the endoluminal viewer’s top legend giving the CCW rotation instruction and current distance from the ROI. IN addition, the Endoluminal Viewer plays a VB-view movie of the rotation from
I
C
T
Θ
1
to
I
C
T
Θ
1
T
. The 3-D airway tree viewer’s display does not change, because the virtual bronchoscope does not advance during this action. The physician now mimics the suggested CCW rotation, as shown in the second endoluminal viewer display in Fig.4(b). The other GUI views, which follow the virtual broncho scope’s movements, do not change. The first approximate real-space/virtual-space synchronization per(5) has now been completed between the guiding virtual bronchoscope and the physician’s “real” bronchoscope. The physician next performs the necessary steps to complete the planned flex-advance maneuvers associated with branch
b
1
. To do this, the physician invokes the “advance” foot-switch command, resulting in the following actions by the three tools [see Fig. 4(c)]. The navigation plan tool highlights the arrive column view
I
C
T
Θ
2
and destination distance (63.6mm) in row 2 for
b
2
. The endoluminal viewer completes the following actions: 1) The top legend gives the “Move Up” instruction and destination distance 64mm. 2) The first video/VB-view pair plays a VB-view advance movie starting with
I
C
T
Θ
1
T
and concluding with
I
C
T
Θ
2
(right). It also continues displaying the current live bronchoscope video stream view (left). As shown, the physician has held the bronchoscope stead from the previous rotation. 3) It creates a second frozen “base-camp” view pair, depicting the previously synchronized vide/VB-view pair after the first rotation [i.e., bottom of Fig. 4(b)]. This pair is enclosed in a green box and labeled “Base Camp
-
Gen 1” to point out the last recorded registered position from the generation-1 branch
b
1
. The 3-D airway-tree viewer plays a movie showing the movement of the virtual bronchoscope from its old pose at
Θ
1
T
to its new pose
Θ
2
situated 64mm from the destination. The physician now flexes and advances the bronchoscope as instructed, to complete the planned route-flex-advance maneuvers associated with
b
1
. Subsequent maneuvers for branches
b
2
,
b
3
,
…
, follow a similar sequence. The online supplement gives a video demonstrating the complete guidance process for Fig. 4.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that the display stops at each upstream and downstream bifurcation position
I
C
T
Θ
1
between the advance command and the rotate command. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 4.
12. Regarding Claim 5, a combination of Khare and Ozaki teaches the apparatus of claim 1. Khare disclose wherein the positional relationship is defined by a first distance that is a distance between the plurality of positions ([pg. 2796, col. 2, A. Conceptual Elements, par. 2, ln. 1 to pg. 2797, col. 1, par. 2, ln. 12] see Euclidean distance for branch length
l
j
). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 5.
13. Regarding Claim 7, a combination of Khare and Ozaki teaches the apparatus of claim 1. Khare further discloses wherein the display aspect includes a rotational display aspect in which the moving image is rotationally displayed around a rotation axis determined for the moving image according to the positional relationship ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5] “Lines 7–26 of Algorithm 1 derive the rotate-flex-advance maneuvers for branch
b
j
. Function VB-view-compute(.) computes a VB view
I
C
T
Θ
j
observing branch-terminating site
v
j
,
w
j
(the view is actually computed at a location 80% along branch
b
j
’s length), while branch-rotation(.,.) compute and angle
θ
∈
[
-
90
°
,
270
°
]
. Angle
θ
establishes the local angular relationship between
b
j
and successor/child branch
b
j
+
1
, i.e., whether
b
j
+
1
appears in the right-half or left-half of the u-v view plane of
I
C
T
Θ
j
.
θ
is computed by first forming a line between child branch
b
j
+
1
’s lumen-region center and the opposing child branch’s lumen center. Next, this line is projected to
I
C
T
Θ
j
’s u-v view plane and centered about the origin. Angle
θ
equals the angle this line forms with
I
C
T
Θ
j
’s positive u-axis. (Note that
θ
could also be derives by a cross-product between appropriate tangent vectors from view sites constituting
b
j
and
b
j
+
1
[16],[30].) Given angle
θ
, angles
θ
1
and
θ
2
are computed, where
-
180
°
≤
-
θ
1
<
0
°
and
0
°
≤
θ
2
<
180
°
. Finally,
θ
1
,
θ
2
, and the current wrist position
α
are combined to derive the maneuvers
{
T
j
,
M
j
}
, where
-
θ
1
signifies a CCW twist,
θ
2
signifies a CW twist, and
M
j
denotes the flex direction. The assigned
T
j
satisfies the wrist-action constraint, such that the downstream bifurcation leading to
b
j
+
1
appears north-south after rotating the bronchoscope an amount
T
j
. In this way, the physician can flex the bronchoscope straight up or down per
M
j
and then advance into the next branch.”, [pg. 2800, Fig. 4] see (a) and (b), [pg. 2801, col. 1, par. 3, ln. 1-13] “Thus, a rotate-flex-advance sequence for
b
j
is represented as follows. During guidance, a green box highlights the VB view corresponding to the current location of interest. The “Arrive” column VB view corresponds to the beginning guidance position near bifurcation
v
j
,
w
j
. CW and CCW instructions in between a view pair indicates the necessary bronchoscope rotation. The “Advance” column VB view visually depicts the result of the preplanned rotation. Because the view clearly shows the desired position after the rotation, we do not explicitly give the rotation angle
T
j
. Next, by way of the blue line corresponding to r, the “Advance” column’s VB view visually indicates whether to flex the bronchoscope up or down when advancing the bronchoscope to branch
b
j
+
1
.”, [pg. 2801, col. 2, 1) VB-rotate-movie, par. 1, ln. 1-6] “VB-rotate-movie (
I
C
T
Θ
j
,
T
j
)
→
I
C
T
Θ
j
T
plays a VB-view movie of the Endoluminal Viewer showing current “Arrive” VB view
I
C
T
Θ
j
rotating an amount
T
j
and stopping at “Advance” view
I
C
T
Θ
j
T
. This function also causes the system state (i.e., the virtual bronchoscope) to move to the row j “Advance” view of the navigation plane.”). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 7.
14. Regarding Claim 11, a combination of Khare and Ozaki teaches the apparatus of claim 7. Khare further disclose wherein the moving image shows an aspect in which the inside of the bronchus is observed from a viewpoint in the bronchus ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5], [pg. 2800, Fig. 4] see (a) and (b), [pg. 2801, col. 1, pare. 3, ln. 1-13]), and the rotational display aspect is an aspect in which the moving image is rotationally displayed around the rotation axis in response to movement of the viewpoint between the plurality of positions ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5], [pg. 2800, Fig. 4] see (a) and (b), [pg. 2801, col. 1, pare. 3, ln. 1-13]), to rotate an opening portion image region showing an opening portion of a bifurcation included in the bronchus toward an upper side in a front view of the display region in which the moving image is displayed ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5] see specifically north-south orientation after rotation, [pg. 2800, Fig. 4] see (a) and (b), [pg. 2801, col. 1, pare. 3, ln. 1-13]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 11.
15. Regarding Claim 12, a combination of Khare and Ozaki teaches the apparatus of claim 11. Khare further discloses wherein the rotational display aspect is an aspect in which the opening portion image region is rotated by a rotation amount corresponding to an angle about the rotation axis between a position of the opening portion image region and a target position on the upper side in the front view ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5], [pg. 2800, Fig. 4] see (b), specifically rightmost image with blue path indicating target branch in north-south orientation, [pg. 2801, col. 1, pare. 3, ln. 1-13]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize
θ
computations and
T
j
of Khare to be analogous to the rotation amount corresponding to an angle about the rotation axis (i.e., the u-axis) between a position of the opening portion image region and a target position on the upper side in the front view. This is because by rotating the image by
T
j
, you effectively obtain the image such that
b
j
+
1
(i.e., the target branch for further navigation) appears in a north-south orientation, though Khare also discloses wherein
b
j
+
1
can be in the lower portion of the image (see Fig. 4, left images, second row and last row). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 12.
16. Regarding Claim 13, a combination of Khare and Ozaki teaches the apparatus of claim 12. Khare further discless wherein the rotational display aspect is an aspect in which the moving image is rotationally displayed around the rotation axis, to locate the opening portion image region on the upper side in the front view at a timing at which the viewpoint reaches a termination position among the plurality of positions ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5], [pg. 2800, Fig. 4] see (b), specifically rightmost image with blue path indicating target branch in north-south orientation, [pg. 2801, col. 1, pare. 3, ln. 1-13], [pg. 2802, col. 2, par. 2, ln. 1 to pg. 2803, col. 1, par. 2, ln. 8]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Khare discloses wherein the rotational display aspect is displayed at a timing at which the viewpoint reaches a termination position, specifically when the operator stops at a given branch
b
1
to invoke a rotation command to determine the next branch along the path r that is to be taken. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 13.
17. Regarding Claim 17, Khare discloses an endoscope apparatus comprising an endoscope that acquires an image showing an aspect of the inside of the bronchus by imaging the inside of the bronchus and that outputs the acquired images ([pg. 2795, col. 2, II. Methods, par. 1, ln. 1-6], [pg. 2796, col. 1, 2) Image-Guided Bronchoscopy, par. 1, ln. 1-14], [pg. 2796, col. 2, A. Conceptual Elements, par. 1, ln. 18], [pg. 2797, Fig. 1] see endoscope, [pg. 2798, Fig. 2] see virtual bronchus image of inside of the bronchus, [pg. 2800, Fig. 4] see both virtual and live video stream of interior of bronchus, wherein the live video is acquired via endoscope). With regard to the remainder of the claim language “the image processing apparatus according to claim 1;”, a combination of Khare and Ozaki teaches the apparatus according to claim 1. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the apparatus and endoscope of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 17.
18. Regarding Claim 18, the claim language is analogous to claim 1 with the exception of “An image processing method:” wherein the reminder of the claim is analogous to claim 1. Khare specifically disclose an image processing method ([pg. 2794, col. 1, Abstract, par. 1, ln. 11-16] “First, it incorporates a new procedure-planning method that automatically computes airway navigation plans conforming to the physician’s bronchoscopy training and manual dexterity. Second, it incorporates a guidance strategy for bronchoscope navigation that enables user-friendly system control via a foot switch, coupled with a novel position-verification mechanism.”). Arguments analogous to claim 1 are further applicable to the remainder of claim 18 in view of the method of Khare. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the method of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 18.
19. Regarding Claim 19, the claim language is analogous to claim 1, with the exception of “A non-transitory computer-readable storage medium storing a program executable by a computer to perform a process comprising”, wherein the remainder of the claim is analogous to claim 1. Khare specifically disclose a non-transitory computer readable storage medium ([pg. 2795, col. 2, II. Methods, par. 1, ln. 1-6]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, t0 combine the non-transitory computer-readable storage medium of Khare with the instruction performed by voice of Ozaki to obtain the invention as specified in claim 19.
20. Claim 6 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over “Hands-Free System for Bronchoscopy Planning and Guidance” to Khare, in view of U.S. Patent No. 7,951,070 to Ozaki, and further in view of JP-2015123250-A to Sakamoto et al. (hereinafter Sakamoto).
21. Regarding Claim 6, a combination of Khare and Ozaki teaches the apparatus of claim 5. Khare discloses wherein the display aspect includes an aspect in which the display of the moving image is advanced {at a speed} determined according to {a time required for} movement between the plurality of positions and the first distance ([pg. 2802, col. 2, par. 2, ln. 1 to pg. 2803, col. 1, par. 2, ln. 8]). Khare does not specifically disclose wherein the advancement speed is according to a time required for movement between the plurality of positions and the first distance, though one of ordinary skill in the art, would recognize that Khare does disclose wherein the image is advanced according to movement between the plurality of positions. Furthermore, Ozaki does not specifically disclose wherein the display aspect includes an aspect in which the display of the moving image is advanced at a speed determined according to a time required for movement between the plurality of positions and the first distance.
However, Sakamoto teaches wherein the image is advanced at a speed determined according to a time required for movement between the plurality of positions and the first distance ([par. 0035, ln. 1-7] “The medical image generation device 1 according to the present invention causes the display device 2 to display the series of virtual endoscopic images generated in step S50 in association with the movement position of the viewpoint. As the movement mode of the viewpoint at that time, a mode in which the operation means 3 is operated to move manually and a mode in which the movement is automatically made at the speed set by the control unit (moving speed setting means 24) are selectable… In the following description, among the movement modes of the two viewpoints, an explanation will be made centering on the mode of moving automatically.”, [par. 0036, ln. 1-9] “As in the conventional configuration, when a virtual endoscopic image is displayed by automatically moving the viewpoint at a constant speed, the viewpoint is such that a portion with a narrow intestinal diameter (a portion where large intestines fold in the core line direction) can be accurately observed. If the moving speed of is set low, the moving speed of the viewpoint of the thick part of the intestinal tract diameter (the part where the large intestine fold extends in the direction of the core line) may be too slow. Therefore, it takes time to display a virtual endoscopic image over the entire large intestine 30, and it becomes difficult to efficiently diagnose. On the contrary, if the moving speed of the viewpoint is set high in accordance with the thick part of the intestine diameter, diagnosis can be made quickly, but the moving speed of the viewpoint becomes too fast in the narrow part of the intestinal diameter, the accuracy It becomes difficult to observe well.”, [par. 0037, ln. 1-7] “Because of this, in order to improve the diagnostic efficiency while securing the observation accuracy, the moving speed of the viewpoint is increased in the thick part of the intestinal diameter, and conversely, the moving speed of the viewpoint is decreased in the thin part of the intestinal diameter Thus, a configuration for displaying a virtual endoscopic image is desired. Therefore, in the medical image generation apparatus 1 according to the present invention, the moving speed of the viewpoint at each position on the core line is set based on the intestine diameter calculated in the program S40 (moving speed setting step of the viewpoint; See S60).”, [par. 0040, ln. 1-13] “FIG. 6 (a) shows the large intestine 30 of the intestinal tract diameter R1 before expansion, and if the linear function of FIG. 5 (a) is applied to the portion of the intestinal tract diameter R1, the viewpoint movement speed of this portion is V1. Set to Then, for example, virtual endoscopic images G (total of 30 sheets) corresponding to the viewpoint positions in the range from the cross section 30a to the cross section 30b are displayed as a moving image in one second. On the other hand, it is assumed that the portion (sections 30a to 30b) of the intestinal tract diameter R1 is expanded to the intestinal tract diameter R2 (⟩ R1) by injecting the medical gas into the large intestine 30. By substituting the intestinal tract diameter R2 into the linear function of FIG. 5A, the viewpoint moving speed of this portion is set to V2 (⟩ V1). As described above, by setting the viewpoint moving speed V according to the intestinal tract diameter R, the virtual endoscopic image G corresponding to the viewpoint position in the range from the cross section 30a to the cross section 30b after expansion as well as before expansion A total of 30 images will be displayed as animation in one second. For this reason, even in the portion where the intestinal tract is expanded, it is possible to perform display with less discomfort without causing the observer to feel a decrease in the viewpoint movement speed.”, [par. 0042, ln. 1-9] “Of the large intestine 30, the descending colon 35 and the sigmoid colon 36, which have a relatively small intestinal diameter, are displayed at a relatively low viewpoint movement speed, so even if the large intestines H are densely packed, the virtual endoscopic image Can be observed carefully, and the observation accuracy can be ensured. On the other hand, since the ascending colon 33 having a relatively large intestine diameter in the large intestine 30 is displayed at a relatively high viewpoint movement speed, the large intestine fold H located away in the core line direction can be efficiently observed… Further, by setting the viewpoint moving speed according to the diameter of the intestine in this way, the observer can be made to feel as if the viewpoint is moved at a constant speed regardless of the interval between the large intestines H, and the observer You can reduce the discomfort you feel.”). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Khare, Ozaki, and Sakamoto as within the same field of image processing for endoscopy, and as analogous to the claimed invention. The motivation to combine is disclosed in Sakamoto, wherein it reduces viewing discomfort, an increase operation speed, and viewing accuracy ([par. 0036, ln. 1-9], [par. 0040, ln. 1-13], [par. 0042, ln. 1-9]). The examiner specifically notes that Sakamoto changes the speed according to the time required to observe the given track of intestine ([par. 0036, ln. 1-9]), and one of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize this is applicable to bronchi as well, since the tree like structure of the bronchi would result in multiple, variable distances between bifurcations that would require more or less time to examine and navigate. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the apparatus of Khare with the instruction performed by voice of Ozaki, and further combined the apparatus of the combination of Khare and Ozaki with the speed according to a time required for inspection/movement of Sakamoto, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the apparatus of Khare with the instruction performed by voice of Ozaki and the speed according to a time required for inspection/movement of Sakamoto to obtain the invention as specified in claim 6.
22. Regarding Claim 9, a combination of Khare and Ozaki teaches the apparatus of claim 7. Khare further discloses wherein {a speed at which} the moving image is rotationally displayed is determined according to the positional relationship ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5], [pg. 2800, Fig. 4] see (a) and (b), [pg. 2801, col. 1, pare. 3, ln. 1-13], [pg. 2801, col. 2, 1) VB-rotate-movie, par. 1, ln. 1-6], [pg. 2802, col. 2, par. 2, ln. 1 to pg. 2803, col. 1, par. 2, ln. 8]). Khare does not specifically disclose wherein a speed of the rotationally displayed image is according to a positional relationship, though one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Khare discloses wherein the rotationally displayed moving image is determined according to the positional relationship. Ozaki does not specifically disclose wherein a speed at which the moving image is rotationally displayed is determined according to the positional relationship.
However, Sakamoto specifically discloses wherein the speed at which the moving image is displayed is determined according to the positional relationship ([par. 0035, ln. 1-7], [par. 0036, ln. 1-9], [par. 0037, ln. 1-7], [par. 0040, ln. 1-13], [par. 0042, ln. 1-9]). Specifically, the motivation to combine remains analogous to claim 6. One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize that the viewer discomfort would likewise result as a response too fast and/or slow a rotation with regards to the “rotate” command movie of the combination of Khare and Ozaki. Specifically, given that the degree of rotation is already measured in Khare ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5], [pg. 2801, col. 1, par. 3, ln. 1-13], [pg. 2801, col. 2, 1) VB-rotate-movie, par. 1, ln. 1-6], [pg. 2802, col. 2, par. 2, ln. 1 to pg. 2803, col. 1, par. 2, ln. 8]), it would have been apparent to one of ordinary skill in the art, before the effective filling date of the claimed invention, that analogous to the speed control for the distance of a given branch for the “advance” movie, you would likewise perform rotation speed control for the angle
θ
for the movie as a result of the “rotate” command (e.g., the larger the degree
θ
, the faster the rotation, the smaller
θ
, the lower the speed of rotation). This would reduce the degree of discomfort experience by the operator, and furthermore, prevent possible confusion with regards to the rotation performed (e.g., if a small and large rotation have the same speed, the small rotation may go unnoticed if the operator is not currently looking at the movie, and/or the large rotation may take too long and the operator may experience discomfort at having to wait for the rotation to finish). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the apparatus of Khare with the instruction performed by voice of Ozaki, and further combined the apparatus of the combination of Khare and Ozaki with the speed according to a time required for inspection/movement of Sakamoto as applied to the angle of rotation of the combination of the apparatus of Khare and Ozaki, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the apparatus of Khare with the instruction performed by voice of Ozaki and the speed according to a time required for inspection/movement of Sakamoto to obtain the invention as specified in claim 9.
23. Claim 8, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over “Hands-Free System for Bronchoscopy Planning and Guidance” to Khare, in view of U.S. Patent No. 7,951,070 to Ozaki, and further in view of “System for Analyzing High-Resolution Three- Dimensional Coronary Angiogram” to Higgins et al. (hereinafter Higgins).
24. Regarding Claim 8, a combination of Khare and Ozaki teaches the apparatus of claim 7. Khare does not specifically disclose wherein the rotation axis is obtained by thinning the bronchial image. Likewise, Ozaki does not specifically disclose wherein the rotation axis is obtained by thinning the bronchial image.
However, Higgins specifically discloses wherein the central axis of a tree-like structure in the body is determined by thinning the image ([pg. 380, col. 1, par. 2, ln. 1 to col. 2, par. 3, ln. 18] “3-D Skeletonization [12], [13]: 3-D skeletonization computes the skeleton of the remaining arterial tree branches and represents the first pass at computing the central axes of the coronary arteries. The desired attributes for the skeleton are [11], [16]: 1) preserve the homotopy of the arterial branches; 2) approximate the central (medial) axes of the branches; 3) be one voxel thick and 4) do not excessively shrink a skeletal component belonging to a branch (i.e., preserve the branch’s shape). We use an iterative skeletonization procedure that assumes 26-connectivity, preserves branch endpoints and draws upon the voxel-deletability procedure of Lobregt et al. [ 171, [ 181. The procedure of Lobregt et al. is very fast, but does not always preserve the homotopy of structures [ll], [19], [20]. While the method is sufficient for our needs, other better 3-D skeletonization procedures exist [19], [20]. 3) Skeleton Pruning: At this stage, v, now contains a 3-D skele- ton of the arterial-tree branches. Unfortunately, the skeleton can contain many useless short branches. Short branches arise in the skeleton for two reasons. First, the angiogram’s limited spatial resolution causes thin short branches to be represented by a small number of voxels. Second, the unavoidable asymmetry in deleting voxels during 3-D skeletonization can cause spurious short branches, emanating from large branches, to form. Since short branches are distracting when making measurements, they are pruned. Skeleton pruning uses the following procedure [12], [13], [22]: 1) erode
l
m
i
n
-
1
endpoints from the skeletal branches and 2) grow back
l
m
i
n
-
1
branch endpoints, conditioning the growth on the original unpruned skeleton. These operations will prune most branches shorter than a preset length
l
m
i
n
. But if two or more branches
≤
l
m
i
n
-
2
voxels in length meet at a bifurcation point which in turn is attached to a longer branch, they will not be deleted… 4) Segment Generation: The branches of the pruned skeleton can exhibit large local directional changes from one voxel to the next. This is because the image’s limited resolution is ill-suited to defin- ing accurate central axes for thin structures such as the coronary arteries. Abrupt local directional changes in a skeletal branch can cause anomalous results in subsequent analyzes. Our remedy to this problem, while imperfect, is simple and effective. Basically, we force each branch to be made up of straight line segments of a preset minimum length
s
m
i
n
. To do this, we form a list of 3-D coordinates for each skeletal branch. For a particular branch, the first voxel on this list will correspond to the beginning voxel of the skeletal branch. The second voxel on the list is then found by traveling
s
m
i
n
-
1
voxels along skeletal branch, etc. If while traveling down a branch, we encounter a bifurcation point, this voxel ends the list. Typically, a branch begins at a bifurcation point, defined as a voxel
v
a
that has three or more 26-neighbors: i.e., three or more skeletal branches pass through the voxel-one mother branch and two or more daughter branches. The computed lists constitute the final defined central axes.”). The examiner specifically notes that while Higgins describes the central axes determination with relation to an arterial pathway, it is specifically noted in Higgins that it has also been applied to bronchial passages for endoscope simulation ([pg. 385, col. 1, par. 1, ln. 2-4] “We have used much of the methodology described here for endoscopy simulation in the bronchial passages using high-resolution 3-D pulmonary images [29].”). This further would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, in view of the similarity between the structures of the bronchi and arterial pathways, specifically, in that they’re both effectively complex tree structures consisting of multiple bifurcations. One of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize that the u-axis of the combination of Khare and Ozaki is analogous to the central axis of the pathway as disclosed in Higgins. The motivation to combine is disclosed in Higgins, wherein it significantly reduces the amount of time to construct and analyze a 3-D tree structure ([pg. 384, col. 1, par. 2, ln. 1-2] “The system reduces the operator’s total interaction time to analyze a typical 3-D DSR angiogram from several hours to several minutes.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the apparatus of Khare with the instruction performed by voice of Ozaki, and further combined the apparatus of the combination of Khare and Ozaki with the rotational axis via thinning of Higgins, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the apparatus of Khare with the instruction performed by voice of Ozaki and the rotational axis via thinning of Higgins to obtain the inventio as specified in claim 8.
25. Regarding Claim 15, a combination of Khare and Ozaki teaches the apparatus of claim 1. Rejections analogous to claim 8 are further applicable to claim 15. Specifically, Khare discloses wherein the moving image shows an aspect in which the inside of the bronchus is observed along a pathway ([pg. 2799, col. 1. par. 5, ln. 1 to pg. 2800, col. 1, par. 1, ln. 5], [pg. 2800, Fig. 4] see (a) and (b), [pg. 2801, col. 1, pare. 3, ln. 1-13]) {obtained by thinning the bronchial image}. Khare does not specifically disclose wherein the image is obtained by thinning the bronchial image. Likewise, Ozaki does not specifically disclose the moving image shows an aspect in which the inside of the bronchus is observed along a pathway.
However, Higgins specifically discloses wherein the inside of a tree-like structure is observed along a pathway obtained by thinning the bronchial image ([pg. 382, Fig. 5 (b) see cross sectional views], [pg. 380, col. 1, par. 2, ln. 1 to col. 2, par. 3, ln. 18], [pg. 381, col. 2, par. 2, ln. 1-12] “The Artery Display has three operating modes: 1) branch viewing; 2) segment viewing; and 3) geometric analysis. Fig. 5 depicts a complete display for branch-viewing mode. Superimposed on each projection image are the previously computed 3-D central axes. While the results of this paper are in black and white, the actual display uses color. In branch-viewing mode, the operator selects an axis segment. The Artery Display then highlights the entire branch from the selected segment back to the bifurcation point spawning the branch [see Fig. 5(a)]. The Artery Display then computes (using bilinear interpolation) and displays a series of 64 x 64 cross-sectional images (actually 4 x blow-ups of 16 x 16 areas) perpendicular to the highlighted branch [see Fig. 5(b)].”). The motivation to combine remains analogous to claim 8. Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize that the u-axis of the combination of Khare and Ozaki is analogous to the central axis of the pathway as disclosed in Higgins, and thus that the cross-sectional images of the apparatus of the combination of Khare and Ozaki could likewise be extracted using the central axis determined via thinning as taught in Higgins. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the apparatus of Khare with the instruction performed by voice of Ozaki, and further combined the apparatus of the combination of Khare and Ozaki with the rotational axis via thinning of Higgins, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the apparatus of Khare with the instruction performed by voice of Ozaki and the rotational axis via thinning of Higgins to obtain the inventio as specified in claim 15.
26. Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over “Hands-Free System for Bronchoscopy Planning and Guidance” to Khare, in view of U.S. Patent No. 7,951,070 to Ozaki, in view of “System for Analyzing High-Resolution Three- Dimensional Coronary Angiogram” to Higgins, and further in view of JP-2015123250-A to Sakamoto.
27. Regarding Claim 16, a combination of Khare, Ozaki, and Higgins teaches the apparatus of claim 15. Khare discloses wherein the display aspect includes an aspect in which the display of the moving image is advanced {at a constant speed} along the pathway between the plurality of positions ([pg. 2802, col. 2, par. 2, ln. 1 to pg. 2803, col. 1, par. 2, ln. 8]). Khare does not specifically disclose this is at a constant speed. Likewise, Ozaki and Higgins do not specifically disclose wherein the display aspect includes an aspect in which the display of the moving image is advanced at a constant speed along the pathway between the plurality of positions.
However, Sakamoto specifically teaches wherein the speed may be set to a constant speed ([par. 0035, ln. 1-7], [par. 0036, ln. 1-9], [par. 0042, ln. 1-9]). Specifically, Sakamoto notes that this is the standard virtual model navigation scheme ([par. 0004, ln. 1-4] “By the way, the virtual endoscopic display is performed by displaying a virtual endoscopic image corresponding to the viewpoint position set on the core line. As a movement form of this viewpoint, the viewpoint is manually operated along the core line. A mode in which the viewpoint is moved and a mode in which the viewpoint is automatically moved at a constant moving speed along the center line can be considered.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize a constant speed to be an alternative to the speed control as disclosed in Sakamoto, and that while a constant speed may present discomfort to a viewer as described in Sakamoto, it is also easier to implement given that no further computation and/or measurements are required. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the apparatus of Khare with the instruction performed by voice of Ozaki and the rotational axis via thinning of Higgins, and further combine the apparatus of the combination of Khare, Ozaki, and Higgins with a constant speed advancement as taught in Sakamoto, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the apparatus of Khare with the instruction performed by voice of Ozaki, the rotational axis via thinning of Higgins, and the constant speed advancement as taught in Sakamoto to obtain the inventio as specified in claim 16.
28. Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over “Hands-Free System for Bronchoscopy Planning and Guidance” to Khare, in view of U.S. Patent No. 7,951,070 to Ozaki, and further in view of U.S. Publication No. 2020/0143594 to Lal et al. (hereinafter Lal).
29. Regarding Claim 14, a combination of Khare and Ozaki teaches the apparatus of claim 1. Khare and Ozaki do not specifically disclose wherein the processor does not receive the instruction or ignores the received instruction during a period in which the moving image is displayed in the display aspect.
However, Lal specifically discloses wherein the processor does not receive the instruction or ignores the received instruction during a period in which another command is being performed ([par. 0020-0026, ln. 1-28] “…command words can be mixed (e.g., “load model please” is the same as “please load model”) except in the case of where a number is needed, then the word before it needs to describe what the number is used for (e.g., “please load image [number]”) is okay but “please load [number] image” is not). In addition, words can be added to the commands but will be ignored (e.g.: “please load image [number] right now” is the same as “please load image [number]”). Also note, the patient number is stored and if the application is closed and reopened that patient is remembered. The number can be a number from 1 through 4 or more. In certain embodiments, the numbers can only be done one at a time. The following are examples of voice-activated input that controls the surgical system 100—along with the system's 100 output: “Please load model”—loads a 3D model. “Please rotate x [number]”—rotates 3D model on the x-axis. “Please rotate y [number]”—rotates 3D model on the y-axis. “Please rotate z [number]”—rotates 3D model on the z-axis. “Please load patient [number]”—Tells the application to switch to a different patient. “Please study [number]”—Displays an animation of a predetermined range of slices and/or displays the animation in either zoomed out or zoomed in mode, depending on the current setting. The study number is displayed.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize the combination of Khare, Ozaki, and Lal as within the same field of medical imaging systems with off-hands commands, and Ozaki and Lal as within the same field of voice commands, and as analogous to the claimed invention. The motivation to combine is disclosed in Lal, wherein it prevents surgeons feeling rushed, possible contaminants, and removes the need for memorization ([par. 0006, ln. 1-15] “The software application is adapted to be a live reference for surgeons to selectively control through natural language commands. For example, the surgeon can verbally request animation of the slices/images for particular study during surgery, in real time. Without this invention, a surgeon, if their memory is unclear, would have to resort to leaving the operating table to reference the patient's information (tomographic, x-ray, et al.). This could impact the patient in longer anesthesia time, surgeon mistakes by feeling rushed, as well as add the possibility of contaminants. Because the software application is adapted for the surgeon to call up a 3D reference model of an organ and rotate it to get the best view possible, the present invention enhances recall of the surgeon where previously he/she would have to depend on memory before.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize the “advance” command of the apparatus of the combination of Khare and Ozaki to be analogous to a voice command as taught in Lal. Specifically, given that Lal discloses wherein only one command may be performed at a time, and the “advance” command results in the images moving from one bifurcation to another as a movie ([Khare, pg. 2802, col. 2, par. 2, ln. 1 to pg. 2803, col. 1, par. 2, ln. 8]), one of ordinary skill in the art, before the effective filling date of the claimed invention, would specifically recognize that during the period of the movie Lal teaches to ignore further commands. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the apparatus of Khare with the instruction performed by voice of Ozaki, and further combined the apparatus of the combination of Khare and Ozaki with the rotational axis via thinning of Higgins, through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the apparatus of Khare with the instruction performed by voice of Ozaki and the ignoring of commands of Lal to obtain the inventio as specified in claim 15.
Allowable Subject Matter
30. Claim 10 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
With regards to claim 10, the examiner notes that the references of record fail to specifically disclose wherein the rotational speed is determined based on a second distance and time required for movement. This is opposed to claim 9, wherein the positional relationship is not defined in such a manner to be attributed to a second distance. The examiner specifically notes that as compared to claim 9, which is obvious in view of the teachings of Sakamoto as applied to the “rotation” movie of the combination of Khare and Ozaki, claim 10 requires a further modification that would necessitate control of the rotation speed according to the distance of the bronchi as opposed to simply applying the teachings of Sakamoto to the angle
θ
of the combination of Khare and Ozaki. This is likewise applicable to claim 6, since while the teachings of Sakamoto may be applied to the distance of the bronchi, Sakamoto, Khare, and Ozaki fail to specifically disclose wherein the distance may also affect the rotation speed. In other words, while it may have been obvious to one of ordinary skill in the art to simply apply the teachings of Sakamoto to distance and rotation individually, since they both would achieve the same affect (i.e., reduction of viewing discomfort for the “advance” and “rotation” movie), it would not have been obvious to have applied the teachings of Sakamoto to the distance and time required for movement (i.e., the “advance” movie) and subsequently further based the rotation speed off of that positional relationship (i.e., the “rotation” movie).
Conclusion
31. The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. See PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAULO ANDRES GARCIA whose telephone number is (703)756-5493. The examiner can normally be reached Mon-Fri, 8-4:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached on (571)272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAULO ANDRES GARCIA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights