DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21-24, 28, 31-34, and 38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1).
Regarding claims 21 and 31, Fujie teaches a surgical system, comprising:
a detector (12 and 13 of fig. 1) comprising a photosensitive site to detect light reflected by a surgical environment that comprises a surgical device (V1, S1, and S2 of Fig. 2 (A); [0018]; a surgical device, S1 and S2 of figs. 1 and 2 (A), 2 (B), 2 (D), and 2 (F); [0020] an operating field image V1 (refer to FIG. 2 (A)) to be a main image composing an endoscopic image of a treating region that an operator looks at in the surgery, and a single lens completing-purpose endoscope 17 (completing-purpose imaging device) for obtaining a completing image V2 (refer to FIG. 2 (B)) for completing the operating field image; [0024] the three-dimensional position measuring device 13 includes markers 22, at least three of which are attached to members to be subjected to position measurement, and a body 23 including light receiving parts 23A for receiving infrared rays emitted by the markers 22), wherein the detector is configured to generate first image data representative of the surgical environment ([0024] the detector detects the three-dimensional positions of the members to be subjected to the position measurement; [0025] It is thereby possible to convert the sets of three-dimensional coordinates of the set points P in the stereo camera coordinate system calculated by the distance measuring device 12 into the sets of three-dimensional coordinates in the reference coordinate system on the basis of the measurement result from the three-dimensional position measuring device 13);
a processor (14 of fig. 1) configured to:
detect, in the first image data, a predetermined wavelength ([0024] the infrared rays would obviously have a predetermined wavelength. It is noted that the infrared rays are a type of electromagnetic radiation with wavelengths longer than visible light but shorter than microwaves, commonly associated with heat; [0041] discloses well-known methods may be used in which predetermined colors are applied to the main body parts S2 and the pieces of image information on the operating field image V1 are distinguished on the basis of the colors to identify the cutoff regions. The predetermined colors (red, blue, and green) would obviously have predetermined wavelengths) associated with a marker on the surgical device ([0024] the three-dimensional position measuring device 13, there are used devices having a well-known configuration which can detect the three-dimensional positions of the markers 22 by tracking the infrared rays following the movements of the markers 22; [0025] It is thereby possible to convert the sets of three-dimensional coordinates of the set points P in the stereo camera coordinate system calculated by the distance measuring device 12 into the sets of three-dimensional coordinates in the reference coordinate system on the basis of the measurement result from the three-dimensional position measuring device 13);
determine an obstruction of an underlying tissue in the surgical environment based on the detection of the predetermined wavelength associated with the marker ([0011] the image processing device obtains image information on a cutoff region in the object space that is hidden on the depth side of a member having a known shape by imaging the member in the main image together with the object space;[0020] Note that the completing-purpose endoscope 17 is disposed so as to be enabled to image a depth-side region of the object space that is hidden by surgical instruments S existing in the operating field image V1 imaged by the operating field endoscope 16 in the operating field image V1; [0028] image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S displayed in the operating field image V1; Fig. 2 (A), a portion of K is hidden by S);
in response to the determined obstruction of the underlying tissue ([0011], [0020], and [0028] image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S; Fig. 2 (C), 2 (D) and 2 (E) illustrating the process of determination of obstruction as the image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S),
generate a first image of the surgical environment based on the first image data and second image data representative of the underlying tissue (V3 of fig. 2 (D), [0029] generating a transformed image V3);
generate a second image of the surgical environment based on a modification of a wavelength associated with the first image (Fig. 2 (F), [0029] and [0042] in the composite image generating means 28, a composite image is generated by performing the following mask process. That is, first, as shown in FIG. 2 (E), a mask is generated by extracting the cutoff regions identified in the operating field image V1. Then, ranges of the sets of in-screen coordinates in the transformed image V3 (the drawing (D)) that match ranges of the in-screen coordinates of the cutoff regions in the operating field image V1 are identified as corresponding regions (dotted-lined regions in the drawing (D)) by the generated mask, and the pieces of image information on these corresponding regions are extracted. The pieces of image information on the cutoff regions in the operating field image V1 are thereafter superimposed or replaced with the pieces of image information on the corresponding regions, and the composite image shown in the drawing (F) is thereby generated; [0041] the identification of the cutoff regions is not limited to the above-described method, and well-known methods may be used in which predetermined colors are applied to the main body parts S2 and the pieces of image information on the operating field image V1 are distinguished on the basis of the colors to identify the cutoff regions); and
send instructions to cause the second image to be displayed ([0005] the second image as the composite image is displayed, [0043] The composite image is an image having the operating field image V1 as a base, in which the pieces of image information on the depth sides of the main body parts S2 are completed by the completing image V2 from the completing-purpose endoscope 17 as if the main body parts S2 of the surgical instruments S displayed in the operating field image V1 are made transparent or translucent).
It is noted that Fujie is silent about the modification of at least one wavelength of one or more wavelengths associated with the image.
Ren teaches the modification at least one wavelength of the one of more wavelengths associated with the image ([0030] markers, 114 of fig. 3, with various spectral absorption/emission may also be used, and the various spectral emission would obviously emit light at various wavelengths; markers 114 of fig. 3, illustrating colors, spectral ranges, and spectrum as various wavelengths; [0035] to [0037]the process of the frame image in Red, Green, and Blue (RGB) based on a predetermined color of the marker 114 to determine the orientation of the marker 114 in the frame image, the predetermined color (Red, Green, or Blue) of is at least one of the one or more colors (Red, Green, and Blue) associated with the image), and the Red, Green, and Blue lights obviously have wavelengths, and the image processor may convert the RGB format image frame into a Hue-Saturation-Value (HSV) space. Based on a predetermined color of the marker 114, the image processor 126 may apply criteria to the hue and saturation channels of the HSV image frame that may separate the marker 114 from the background in order to bring out and estimate the image of the marker 114, so this disclosure suggests the modification of the at least one wavelength of one or more wavelengths associated with the image).
Taking the teachings of Fujie and Ren together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify to modify at least one wavelength of one of the more wavelengths of Ren associated with the image of Fujie to provide the accuracy of the position and orientation of the surgical instruments to assist a surgeon during a surgical operation.
Regarding claims 22 and 32, Fujie and Ren teach the surgical system of claim 21, Fujie further teaches wherein the first image data is modified by replacing the first image data with at least a portion of the second image data ([0028] composite image generating means 28 for identifying corresponding regions (dotted-lined regions in FIG. 2 (D)) corresponding to the cutoff regions in the transformed image V3 and for replacing the image information on the cutoff regions in the operating field image V1 with image information on the corresponding regions in the transformed image V3; Figs. 2 (D) and 2 (E) The pieces of image information on the cutoff regions, Fig. 2 (E), in the operating field image V1 are thereafter superimposed or replaced with the pieces of image information on the corresponding regions, dotted line regions of Fig. 2 (D), and the composite image shown in the drawing (F) is thereby generated).
Regarding claims 23 and 33, Fujie and Ren teach the surgical system of claim 21, Fujie further teaches wherein the processor is further configured to: determine an obstructed region (S and S2 of fig. 2 (A) and 2 (E), [0011] a cutoff region in the object space that is hidden on the depth side of a member; [0012] image information on a portion of the object space is hidden by the surgical instrument, [0028] image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S displayed in the operating field image V1) associated with the underlying tissue based on the detection of the predetermined wavelength ([0024] the detected three-dimensional positions of the instruments based on the infrared rays of the markers), wherein the first image is generated by rendering the obstructed region transparent based on the second image data ([0043] The composite image is an image having the operating field image V1 as a base, in which the pieces of image information on the depth sides of the main body parts S2 are completed by the completing image V2 from the completing-purpose endoscope 17 as if the main body parts S2 of the surgical instruments S displayed in the operating field image V1 are made transparent or translucent).
Regarding claims 24 and 34, Fujie and Ren teach the surgical system of claim 21, Fujie further teaches wherein the detector comprises a first image capturing device ([0020] a single lens completing-purpose endoscope 17 (completing-purpose imaging device) for obtaining a completing image V2 (refer to FIG. 2 (B)) for completing the operating field image), wherein the processor is further configured to obtain the second image data from a second image capturing device ([0021]to [0023] stereo camera and the set points P), and wherein the first and second image capturing devices have different vantage points (the single lens completing-purpose endoscope 17 and stereo camera 19 of figure 1 have different views as vantage points, Fig. 2 (A) and Fig. 2 (B)).
Regarding claims 28 and 38, Fujie and Ren teach the surgical system of claim 21, Ren further teaches wherein the predetermined wavelength is associated with a chrominance that is not naturally occurring in the surgical environment ([0035] a predetermined color of marker, 114 of figure 3, indicates the color is not naturally occurring in the surgical environment); wherein the processor is configured to search the first image data to detect the predetermined wavelength ([0035] based on a predetermined color of the marker 114, the image processor 126 may apply criteria to the hue and saturation channels of the HSV image frame that may separate the marker 114 from the background in order to bring out and estimate the image of the marker 114).
Claim(s) 25 and 35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Smith et al. (US 20190289284 A1).
Regarding claims 25 and 35, Fujie and Ren teach the surgical system of claim 24 and method of claim 34. However, Fujie and Ren do not teach wherein the processor is further configured to:
determine a distance between the first image capturing device and the second image capturing device; and
generate a three-dimensional (3D) image of the surgical environment based on the first image data, the second image data, and the distance between the first image capturing device and the second image capturing device.
Smith teaches determine a distance between the first image capturing device and the second image capturing device (Cameras 100 of fig. 1, the cameras 100 are a distance apart); and
generate a three-dimensional (3D) image of the surgical environment ([0027] the use of a separate display for each eye allows for stereoscopic display; [0028] the user can be presented with a stereoscopic image that mimics the perspective that the user would see without wearing the display assembly 200; [0033] the display module 309 can be configured to provide display output information to the display assembly 200 for presentation to the user via the display device 209. As noted above, this can include stereoscopic display, in which different images are provided to each eye via first and second display devices 219a-b (FIG. 2B)) based on the first image data, the second image data ([0031] and [0033] the detection of the markers), and the distance between the first image capturing device and the second image capturing device (100 of fig. 1, the cameras are arranged in a distance).
Taking the teachings of Fujie, Ren, and Smith together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the generation of 3D image based on the distance between camera of Smith into the processor of Fujie and Ren for allowing telesurgery to be performed while providing the remote user with the sense of presence and perspective to improve the surgical visualization ([0038] of Smith).
Claim(s) 26 and 36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Schoepp (US 20120078236 A1).
Regarding claims 26 and 36, Fujie and Ren teach the surgical system of claim 21 and method of claim 31, Fufie further teaches wherein the surgical device is associated with a plurality of markers (22 of fig. 1).
However, Fujie and Ren do not teach wherein the processor is further configured to:
determine first orientation information associated with a first marker of the plurality of markers based on a position of the first marker relative to a position of a second marker of the plurality of markers;
obtain second orientation information associated with the first marker from a database; and compare the first and second orientation information of the first marker to identify the surgical device in the image data.
Schoepp teaches wherein the processor is further configured to:
determine first orientation information associated with a first marker of the plurality of markers based on a position of the first marker relative to a position of a second marker of the plurality of markers (30 of fig. 2, [0021] The optical navigation markers 30 on the tracking element 14 comprise a set of shapes and/or colors that can uniquely identify the position and/or identity of the surgical tool 12 when viewed from any viewing angle by the camera 18);
obtain second orientation information associated with the first marker from a database (52 of fig.1, [0032] the optical navigation markers 30 available from the database 52); and
compare the first and second orientation information of the first marker to identify the surgical device in the image data ([0032] the digital image recognition module 54 is capable of determining the position of the surgical tool 12 from a two-dimensional image of the tracking element 14 visible to the camera 18 by, for example, comparing the image of the optical navigation markers 30 with the form factor data of both the surgical tool 12 and the optical navigation markers 30 available from the database 52).
Taking the teachings of Fujie, Ren, Schoepp together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the comparison of markers of Schoepp into the processor of Fujie and Ren to provide the same advantage of having a known spatial relationship between the fiducial markers and the camera coordinate system, and may further improve ease of handling during imaging and make it easier to maintain sterility during the surgical procedure ([0028] of Schoepp).
Claim(s) 27 and 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) and Schoepp (US 20120078236 A1) as applied to claims 21 and 26, and further in view of Yates et al. (US 20180049829 A1).
Regarding claims 27 and 37, Fujie, Ren, and Schoepp teach the surgical system of claim 26 and method of claim 36, Fujie, Ren, and Schoepp wherein the surgical device comprises a shaft and an end effector provided at a distal end of the shaft, wherein the plurality of markers is a first plurality of markers, wherein the shaft is marked using the first plurality of markers, and the end effector is marked using a second plurality of markers.
Yates teaches wherein the surgical device comprises a shaft (1302 of fig. 9, [0060]) and an end effector provided at a distal end of the shaft (1304 of fig. 9, [0060]), wherein the plurality of markers is a first plurality of markers (1308 of fig. 9, [0062] one or more markers), wherein the shaft is marked using the first plurality of markers (1308 of fig. 9, shaft markers), and the end effector is marked using a second plurality of markers (1310 of fig. 9, the surgical tool marker is further disclosed in figures 10-11, [0065]-[0066]).
Taking the teachings of Fujie, Ren, Schoepp, and Yates together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the markers on the shaft and the markers on the end effector of Yates into the processor of Fujie in view of Ren and Schoepp to provide the accurate locations of each portion of the surgical instrument.
Fujie further teaches wherein the processor is further configured to: based on detecting, in a portion of the first image data (Fig. 2 (E), the white portions of the image based on the three dimensional positions of the markers, [0042]), at least one of the first plurality of markers and not a marker of the second plurality of markers (Fig. 2 (E), the white portions show the portion of the surgical instruments, S2 of fig. 2 (A), not including tips, S1 of fig. 2 (A), of the surgical instruments; [0041] well-known methods may be used in which predetermined colors are applied to the main body parts S2 and the pieces of image information on the operating field image V1 are distinguished on the basis of the colors to identify the cutoff regions; wherein the main body parts S2 are considered as the shaft and marked by the markers as disclosed and modified Yates as mentioned above, Yates: 1312 of fig. 9 and [0060]), replace the portion of the first image data with a portion of the second image data ([0028] and [0042] in the composite image generating means 28, a composite image is generated by performing the following mask process. That is, first, as shown in FIG. 2 (E), a mask is generated by extracting the cutoff regions identified in the operating field image V1. Then, ranges of the sets of in-screen coordinates in the transformed image V3 (the drawing (D)) that match ranges of the in-screen coordinates of the cutoff regions in the operating field image V1 are identified as corresponding regions (dotted-lined regions in the drawing (D)) by the generated mask, and the pieces of image information on these corresponding regions are extracted. The pieces of image information on the cutoff regions in the operating field image V1 are thereafter superimposed or replaced with the pieces of image information on the corresponding regions, and the composite image shown in the drawing (F) is thereby generated).
Claim(s) 29 and 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Bailey et al. (US 20170272732 A1)
Regarding claims 29 and 39, Fujie and Ren teach the surgical system of claim 21 and method of claim 31, Fujie and Ren do not teach wherein the surgical system comprises a display, and wherein the processor is further configured to: shift a first wavelength of the one or more wavelengths associated with the first image to a second wavelength, and wherein the second image is displayed on the display using the second wavelength.
Bailey teaches wherein the surgical system (fig. 2) comprises a display (210 of figs. 2 and 3, [0028] a display screen), and wherein the processor (205 of fig. 2) is further configured to: shift a first wavelength of the one or more wavelengths associated with the first image to a second wavelength ([0018] and [0019] shift the wavelengths), and wherein the second image is displayed on the display using the second wavelength ([0037]).
Taking the teachings of Fujie, Ren, and Bailey together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide the controllable wavelength ranges corresponding to the left eye and right eye frames, which may improve the extinction ratio of the glasses, and improving the extinction ratio may improve the 3D effects and reduce the likelihood of motion sickness.
Claim(s) 30 and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Kessman et al. (US 6379302 B1).
Regarding claims 30 and 40, Fujie and Ren teach the surgical system of claim 21 and method of claim 31, Fujie and Ren do not teach wherein the processor is further configured to generate a third image based on the first image data or the second image data, and wherein the instructions cause the second image to be displayed side by side with the third image.
Kessman teaches wherein the processor is further configured to generate a third image based on the first image data or the second image data, and wherein the instructions cause the second image to be displayed side by side with the third image (Col. 8, lines 47-60).
Taking the teachings of Fujie, Ren, and Schoepp together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the side by side display of the images of Kessman to into the display device of Fujie and Ren to provide more information and better image viewed by surgeon during surgery operation.
Response to Arguments
Applicant's arguments filed 08/13/2026 have been fully considered but they are not persuasive.
The applicant asserts that Fujie and Ren, in combination or alone, do not teach or suggest at least determin[ing] an obstruction of an underlying tissue in the surgical environment based on the detection of the predetermined wavelength associated with the marker and, in response to the determined obstruction of the underlying tissue, generat[ing] a first image of the surgical environment, as recited in independent claim 21.
The examiner strongly disagrees with the applicant. It is submitted that Fujie discloses determining an obstruction of an underlying tissue (K and S of fig. 2(A), a portion of K is hidden by S; [0011], [0020], and [0028] determining the hidden of the underlying tissue) in the surgical environment (V1 of fig. 2(A)) based on the detection of the predetermined wavelength ([0024] and [0041] predetermined colors and infrared rays would obviously have predetermined wavelength) associated with the marker ([0024] markers 22), and in response to the determined obstruction of the underlying tissue ([0011], [0024], and [0028] the determined hidden of the underlying tissue), generating a first image of the surgical environment (V3 of fig. 2(D), [0029] generating a transformed image V3 as a first image of the surgical environment).
The applicant further asserts the Office Action draws a parallel between the infrared emissions in Fujie and the claimed predetermined wavelength (the Office Action, page 3). However, the detection of the infrared emission in Fujie determines the position of the markers associated with surgical instruments, rather than determining an obstruction of underlying tissue as required by claim 21. For example, paragraph [0024] of Fujie explains that the three-dimensional position measuring device detects the three-dimensional positions of markers 22 by tracking infrared rays emitted by the moving markers. Paragraph [0025] further explains
that the resulting marker positions are used to identify three-dimensional coordinates of the surgical instruments. Thus, Fujie uses the infrared emissions to obtain positional information of the markers associated with the surgical instruments.
The examiner strongly disagrees with the applicant. It is submitted that Fujie teaches the detection of the infrared emission to identify the position of the markers associated with surgical instruments ([0024] and [0025]). Fujie further teaches determining an obstruction of underlying tissue ([0011], [0020], and [0028] determine the hidden portion behind the surgical instruments) based on the detected position of the surgical instruments by tracking the movement and illumination of the markers ([0024] and [0025] detect the location of the surgical instrument by tracking the movement and illumination of the markers, [0024] and [0041] detection of the position of the surgical instruments by tracking the infrared rays and predetermined colors of the markers 22).
The applicant further asserts that the rejection does not identify a passage in which Fujie uses detection of an infrared wavelength to determine that tissue is obstructed but relies on passages on identification of a cutoff-region in Fujie to teach this claim feature (Page 4 of the Office Action).
The examiner strongly disagrees with the applicant. It is submitted that Fujie uses the detection of an infrared wavelength ([0024], [0025], and [0041] detection of the position of the instruments by tracking the movement and illumination of the markers 22) to determine that tissue is obstructed by the surgical instruments ([0011], [0020], and [0028] determine the hidden portion behind the surgical instruments based on the detected position of the surgical instruments, [0024], [0025], and [0041]).
The applicant further asserts that the cutoff-region is only positional information used to complete Fujie's imaging operations in paragraph [0041], so Fujie cannot be said to teach least determin[ing] an obstruction of an underlying tissue in the surgical environment based on the detection of the predetermined wavelength associated with the marker and, in response to the determined obstruction of the underlying tissue, generat[ing] a first image of the surgical environment, as recited in independent claim 21.
The examiner strongly disagrees with the applicant. It is submitted that Fujie discloses a determination of an obstruction (Fig. 2 (A), a portion K is hidden by S; Fig. 2(C), 2(D), and 2(E) illustrating the process of the determination of obstruction as the image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S; [0011], [0020], [0028]) based on detection of a predetermined wavelength associated with a marker ([0024], [0025], and [0041] detection of the position of the surgical instruments, where the hidden portion of K is behind the surgical instruments of S, by tracking the movement and illumination of the markers 22).
Fujie further discloses generating a first image of the surgical environment (V3 of fig. 2(D), [0029] generating transformed image V3 that is considered as a first image) based on the first image data ([0024], [0025], and [0041] the detected position of the surgical instrument) and second image data represent of the underlying tissue ([0011], [0020], 0028] the hidden part behind the surgical instruments).
The applicant further asserts that Fujie and Ren, in combination or alone, do not teach or suggest generating a second image based on a modification of a wavelength.
The examiner strongly disagrees with the applicant. It is submitted that Fuji teaches a second image based on a modification of the first image ([0029] and [0042]) that associates with a wavelength ([0041] predetermined wavelengths), and the first image as the transformed image (V3 of fig. 2(D), [0029]) that comprises the image of the surgical instruments and the detected position of the surgical instruments ([0024], 0025], and [0041]), wherein the detected position of the surgical instruments by tracking the movement and illumination of the markers using the emission of infrared rays and predetermined colors ([0024] and [0041]), so the first image is transformed as V3, the locations of the surgical instruments is transformed as modified based on the detected locations, and the detected location of the surgical based on the wavelength.
Ren teaches at least one wavelength of the one or more wavelengths associated with the image (114 of fig. 3, variations wavelengths of the markers, [0030], [0035] to [0037] at least one wavelength of one or more wavelength associated with the image). Ren suggests generating an image ([0039] processing image for a display) based on a modification of a wavelength of at least one wavelength of one or more wavelengths associated with the first image ([0035] the image processor 126 may receive the image frame in Red-Green-Blue (RGB) format. At 404, the image processor 126 may convert the RGB format image frame into a Hue-Saturation-Value (HSV) space). Based on a predetermined color of the marker 114, the image processor 126 may apply criteria to the hue and saturation channels of the HSV image frame that may separate the marker 114 from the background in order to bring out and estimate the image of the marker 114). This disclosure suggests Ren discloses generating an image based on a modification of a wavelength of at least one wavelength of one or more wavelengths associated with the first image ([0035] to [0039]).
It is noted that the patent application publication US 20250331932 A1 discloses the image of the surgical environment can also be modified to replace the images of selected portions of the surgical instrument with those of the surgical environment (e.g., target tissues, tissues surrounding the target tissue, etc.) in paragraphs [0037] and [0057], so this disclosure discloses the claim features as generate a second image of the surgical environment based on a modification of at least one wavelength of one or more wavelengths associated with the first image that would obviously be taught by Fujie (Fig. [0029] and [0042] generating the second image of the surgical environment as the composite image based on the modification of the first image that comprises the location of the surgical instruments detected by the movement and illumination of the markers).
The applicant asserts that Ren does not teach or suggest generating the image based on the modification of the at least wavelength of one or more wavelengths associated with the first image.
The examiner strongly disagrees with the applicant. It is submitted that Ren discloses generating an image ([0039] processing image for display) based on a modification of a wavelength of at least one wavelength of one or more wavelengths associated with the first image ([0035] the image processor 126 may receive the image frame in Red-Green-Blue (RGB) format and convert the RGB format image frame into a Hue-Saturation-Value (HSV) space). Based on a predetermined color of the marker 114, the image processor 126 may apply criteria to the hue and saturation channels of the HSV image frame that may separate the marker 114 from the background in order to bring out and estimate the image of the marker 114). This disclosure suggests Ren discloses generating the image based on the modification of the at least wavelength of one or more wavelengths associated with the first image ([0035] to [0039]).
The applicant further asserts that rather, Ren relies on a predetermined color of the marker to separate the marker from the background (paragraphs [0030] and [0035]-[0037] of Ren). As such, Ren's image-processing pipeline depends on identifying that predetermined color as a fixed reference characteristic for marker extraction. That is, a modification of the color in Ren will disrupt the marker extraction process. Accordingly, Ren cannot be said to teach or suggest generating an image based on the modification of the wavelength, as claimed.
The examiner strongly disagrees with the applicant. It is submitted that Ren discloses processing an image based on the modification of the wavelength ([0035]-[0037] the image processor 126 may receive the image frame in Red-Green-Blue (RGB) format. At 404, the image processor 126 may convert the RGB format image frame into a Hue-Saturation-Value (HSV) space. At 406, after the image frame has been enhanced to bring out the contrast and feature, the image processor 126 may determine a first-order estimation mask of the marker 114. For example, based on a predetermined color of the marker 114, the image processor 126 may apply criteria to the hue and saturation channels of the HSV image frame that may separate the marker 114 from the background in order to bring out and estimate the image of the marker 114; [0038]- [0039] generating and displaying the image frame overlaid with the indicators).
It is noted that the specification of the patent application publication US 20250331932 A1 discloses generate a modified image in paragraphs [0037] and [0057] that is treated as generate a second image of the surgical environment based on a modification of at least one wavelength of one or more wavelengths associated with the first image and the specification discloses a wavelength such as predetermined colors, infrared and ultraviolet light, spectrum, and spectral range in paragraphs [0044], [0046], and [0048]. Therefore, a modification of at least one wavelength of one or more wavelengths associated with the first image is treated as generate a modified image with the position of the markers that are determined by tracking the wavelength of the markers attached on the surgical instrument as taught by Fujie ([0029] transformed image V3).
It is suggested that a modification of at least one wavelength of one or more wavelengths associated with the first image should be further clarified.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TUNG T VO whose telephone number is (571)272-7340. The examiner can normally be reached Monday-Friday 6:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Pendleton can be reached at 571-272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
TUNG T. VO
Primary Examiner
Art Unit 2425
/TUNG T VO/Primary Examiner, Art Unit 2425
Prosecution Insights