CTNF 19/211,913 CTNF 74761 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 21-24, 28, 31-34, and 38 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) . Regarding claims 21 and 31, Fujie teaches a surgical system, comprising: a detector (12 and 13 of fig. 1) comprising a photosensitive site to detect light reflected by a surgical environment that comprises a surgical device (V1, S1, and S2 of Fig. 2 (A); [0018]; a surgical device, S1 and S2 of figs. 1 and 2 (A), 2 (B), 2 (D), and 2 (F); [0020] an operating field image V1 (refer to FIG. 2 (A)) to be a main image composing an endoscopic image of a treating region that an operator looks at in the surgery, and a single lens completing-purpose endoscope 17 (completing-purpose imaging device) for obtaining a completing image V2 (refer to FIG. 2 (B)) for completing the operating field image; [0024] the three-dimensional position measuring device 13 includes markers 22, at least three of which are attached to members to be subjected to position measurement, and a body 23 including light receiving parts 23A for receiving infrared rays emitted by the markers 22), wherein the detector is configured to generate first image data representative of the surgical environment ([0024] the detector detects the three-dimensional positions of the members to be subjected to the position measurement; [0025] It is thereby possible to convert the sets of three-dimensional coordinates of the set points P in the stereo camera coordinate system calculated by the distance measuring device 12 into the sets of three-dimensional coordinates in the reference coordinate system on the basis of the measurement result from the three-dimensional position measuring device 13); a processor (14 of fig. 1) configured to: detect, in the first image data, a predetermined wavelength ([0024] the infrared rays would obviously have a predetermined wavelength. It is noted that the infrared rays are a type of electromagnetic radiation with wavelengths longer than visible light but shorter than microwaves, commonly associated with heat; [0041] discloses well-known methods may be used in which predetermined colors are applied to the main body parts S2 and the pieces of image information on the operating field image V1 are distinguished on the basis of the colors to identify the cutoff regions. The predetermined colors (red, blue, and green) would obviously have predetermined wavelengths) associated with a marker on the surgical device ([0024] the three-dimensional position measuring device 13, there are used devices having a well-known configuration which can detect the three-dimensional positions of the markers 22 by tracking the infrared rays following the movements of the markers 22; [0025] It is thereby possible to convert the sets of three-dimensional coordinates of the set points P in the stereo camera coordinate system calculated by the distance measuring device 12 into the sets of three-dimensional coordinates in the reference coordinate system on the basis of the measurement result from the three-dimensional position measuring device 13); determine an obstruction of an underlying tissue in the surgical environment based on the detection of the predetermined wavelength associated with the marker ([0011] the image processing device obtains image information on a cutoff region in the object space that is hidden on the depth side of a member having a known shape by imaging the member in the main image together with the object space;[0020] Note that the completing-purpose endoscope 17 is disposed so as to be enabled to image a depth-side region of the object space that is hidden by surgical instruments S existing in the operating field image V1 imaged by the operating field endoscope 16 in the operating field image V1; [0028] image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S displayed in the operating field image V1; Fig. 2 (A), a portion of K is hidden by S); in response to the determined obstruction of the underlying tissue ([0011], [0020], and [0028] image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S; Fig. 2 (C), 2 (D) and 2 (E) illustrating the process of determination of obstruction as the image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S), generate a first image of the surgical environment based on the first image data and second image data representative of the underlying tissue (V3 of fig. 2 (D), [0029] generating a transformed image V3); generate a second image of the surgical environment based on a modification of the first image ((Fig. 2 (F), [0029] and [0042] in the composite image generating means 28, a composite image is generated by performing the following mask process. That is, first, as shown in FIG. 2 (E), a mask is generated by extracting the cutoff regions identified in the operating field image V1. Then, ranges of the sets of in-screen coordinates in the transformed image V3 (the drawing (D)) that match ranges of the in-screen coordinates of the cutoff regions in the operating field image V1 are identified as corresponding regions (dotted-lined regions in the drawing (D)) by the generated mask, and the pieces of image information on these corresponding regions are extracted. The pieces of image information on the cutoff regions in the operating field image V1 are thereafter superimposed or replaced with the pieces of image information on the corresponding regions, and the composite image shown in the drawing (F) is thereby generated; [0041] the identification of the cutoff regions is not limited to the above-described method, and well-known methods may be used in which predetermined colors are applied to the main body parts S2 and the pieces of image information on the operating field image V1 are distinguished on the basis of the colors to identify the cutoff regions); and send instructions to cause the second image to be displayed ([0005] the second image as the composite image is displayed, [0043] The composite image is an image having the operating field image V1 as a base, in which the pieces of image information on the depth sides of the main body parts S2 are completed by the completing image V2 from the completing-purpose endoscope 17 as if the main body parts S2 of the surgical instruments S displayed in the operating field image V1 are made transparent or translucent). It is noted that Fujie is silent about at least one wavelength of the one of more wavelengths associated with the image. Ren teaches at least one wavelength of the one of more wavelengths associated with the image ([0030] markers, 114 of fig. 3, with various spectral absorption/emission may also be used, and the various spectral emission would obviously emit light at various wavelengths; markers 114 of fig. 3, illustrating colors, spectral ranges, and spectrum as various wavelengths; [0035] to [0037]the process of the frame image in Red, Green, and Blue (RGB) based on a predetermined color of the marker 114 to determine the orientation of the marker 114 in the frame image, the predetermined color (Red, Green, or Blue) of is at least one of the one or more colors (Red, Green, and Blue) associated with the image), and the Red, Green, and Blue lights obviously have wavelengths). Taking the teachings of Fujie and Ren together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify to modify at least one wavelength of one of the more wavelengths of Ren associated with the image of Fujie to provide the accuracy of the position and orientation of the surgical instruments to assist a surgeon during a surgical operation. Regarding claims 22 and 32, Fujie and Ren teach the surgical system of claim 21, Fujie further teaches wherein the first image data is modified by replacing the first image data with at least a portion of the second image data ([0028] composite image generating means 28 for identifying corresponding regions (dotted-lined regions in FIG. 2 (D)) corresponding to the cutoff regions in the transformed image V3 and for replacing the image information on the cutoff regions in the operating field image V1 with image information on the corresponding regions in the transformed image V3; Figs. 2 (D) and 2 (E) The pieces of image information on the cutoff regions, Fig. 2 (E), in the operating field image V1 are thereafter superimposed or replaced with the pieces of image information on the corresponding regions, dotted line regions of Fig. 2 (D), and the composite image shown in the drawing (F) is thereby generated). Regarding claims 23 and 33, Fujie and Ren teach the surgical system of claim 21, Fujie further teaches wherein the processor is further configured to: determine an obstructed region (S and S2 of fig. 2 (A) and 2 (E), [0011] a cutoff region in the object space that is hidden on the depth side of a member; [0012] image information on a portion of the object space is hidden by the surgical instrument, [0028] image information on cutoff regions in the object space hidden on the depth side thereof by the surgical instruments S displayed in the operating field image V1) associated with the underlying tissue based on the detection of the predetermined wavelength ([0024] the detected three-dimensional positions of the instruments based on the infrared rays of the markers), wherein the first image is generated by rendering the obstructed region transparent based on the second image data ([0043] The composite image is an image having the operating field image V1 as a base, in which the pieces of image information on the depth sides of the main body parts S2 are completed by the completing image V2 from the completing-purpose endoscope 17 as if the main body parts S2 of the surgical instruments S displayed in the operating field image V1 are made transparent or translucent). Regarding claims 24 and 34, Fujie and Ren teach the surgical system of claim 21, Fujie further teaches wherein the detector comprises a first image capturing device ([0020] a single lens completing-purpose endoscope 17 (completing-purpose imaging device) for obtaining a completing image V2 (refer to FIG. 2 (B)) for completing the operating field image), wherein the processor is further configured to obtain the second image data from a second image capturing device ([0021]to [0023] stereo camera and the set points P), and wherein the first and second image capturing devices have different vantage points (the single lens completing-purpose endoscope 17 and stereo camera 19 of figure 1 have different views as vantage points, Fig. 2 (A) and Fig. 2 (B)). Regarding claims 28 and 38, Fujie and Ren teach the surgical system of claim 21, Ren further teaches wherein the predetermined wavelength is associated with a chrominance that is not naturally occurring in the surgical environment ([0035] a predetermined color of marker, 114 of figure 3, indicates the color is not naturally occurring in the surgical environment); wherein the processor is configured to search the first image data to detect the predetermined wavelength ([0035] based on a predetermined color of the marker 114, the image processor 126 may apply criteria to the hue and saturation channels of the HSV image frame that may separate the marker 114 from the background in order to bring out and estimate the image of the marker 114) . 07-21-aia AIA Claim (s) 25 and 35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Smith et al. (US 20190289284 A1) . Regarding claims 25 and 35, Fujie and Ren teach the surgical system of claim 24 and method of claim 34. However, Fujie and Ren do not teach wherein the processor is further configured to: determine a distance between the first image capturing device and the second image capturing device; and generate a three-dimensional (3D) image of the surgical environment based on the first image data, the second image data, and the distance between the first image capturing device and the second image capturing device. Smith teaches determine a distance between the first image capturing device and the second image capturing device (Cameras 100 of fig. 1, the cameras 100 are a distance apart); and generate a three-dimensional (3D) image of the surgical environment ([0027] the use of a separate display for each eye allows for stereoscopic display; [0028] the user can be presented with a stereoscopic image that mimics the perspective that the user would see without wearing the display assembly 200; [0033] the display module 309 can be configured to provide display output information to the display assembly 200 for presentation to the user via the display device 209. As noted above, this can include stereoscopic display, in which different images are provided to each eye via first and second display devices 219a-b (FIG. 2B)) based on the first image data, the second image data ([0031] and [0033] the detection of the markers), and the distance between the first image capturing device and the second image capturing device (100 of fig. 1, the cameras are arranged in a distance). Taking the teachings of Fujie, Ren, and Smith together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the generation of 3D image based on the distance between camera of Smith into the processor of Fujie and Ren for allowing telesurgery to be performed while providing the remote user with the sense of presence and perspective to improve the surgical visualization ([0038] of Smith) . 07-21-aia AIA Claim (s) 26 and 36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Schoepp (US 20120078236 A1) . Regarding claims 26 and 36, Fujie and Ren teach the surgical system of claim 21 and method of claim 31, Fufie further teaches wherein the surgical device is associated with a plurality of markers (22 of fig. 1). However, Fujie and Ren do not teach wherein the processor is further configured to: determine first orientation information associated with a first marker of the plurality of markers based on a position of the first marker relative to a position of a second marker of the plurality of markers; obtain second orientation information associated with the first marker from a database; and compare the first and second orientation information of the first marker to identify the surgical device in the image data. Schoepp teaches wherein the processor is further configured to: determine first orientation information associated with a first marker of the plurality of markers based on a position of the first marker relative to a position of a second marker of the plurality of markers (30 of fig. 2, [0021] The optical navigation markers 30 on the tracking element 14 comprise a set of shapes and/or colors that can uniquely identify the position and/or identity of the surgical tool 12 when viewed from any viewing angle by the camera 18); obtain second orientation information associated with the first marker from a database (52 of fig.1, [0032] the optical navigation markers 30 available from the database 52); and compare the first and second orientation information of the first marker to identify the surgical device in the image data ([0032] the digital image recognition module 54 is capable of determining the position of the surgical tool 12 from a two-dimensional image of the tracking element 14 visible to the camera 18 by, for example, comparing the image of the optical navigation markers 30 with the form factor data of both the surgical tool 12 and the optical navigation markers 30 available from the database 52). Taking the teachings of Fujie, Ren, Schoepp together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the comparison of markers of Schoepp into the processor of Fujie and Ren to provide the same advantage of having a known spatial relationship between the fiducial markers and the camera coordinate system, and may further improve ease of handling during imaging and make it easier to maintain sterility during the surgical procedure ([0028] of Schoepp) . 07-21-aia AIA Claim (s) 27 and 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) and Schoepp (US 20120078236 A1) as applied to claims 21 and 26, and further in view of Yates et al. (US 20180049829 A1) . Regarding claims 27 and 37, Fujie, Ren, and Schoepp teach the surgical system of claim 26 and method of claim 36, Fujie, Ren, and Schoepp wherein the surgical device comprises a shaft and an end effector provided at a distal end of the shaft, wherein the plurality of markers is a first plurality of markers, wherein the shaft is marked using the first plurality of markers, and the end effector is marked using a second plurality of markers. Yates teaches wherein the surgical device comprises a shaft (1302 of fig. 9, [0060]) and an end effector provided at a distal end of the shaft (1304 of fig. 9, [0060]), wherein the plurality of markers is a first plurality of markers (1308 of fig. 9, [0062] one or more markers), wherein the shaft is marked using the first plurality of markers (1308 of fig. 9, shaft markers), and the end effector is marked using a second plurality of markers (1310 of fig. 9, the surgical tool marker is further disclosed in figures 10-11, [0065]-[0066]). Taking the teachings of Fujie, Ren, Schoepp, and Yates together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the markers on the shaft and the markers on the end effector of Yates into the processor of Fujie in view of Ren and Schoepp to provide the accurate locations of each portion of the surgical instrument. Fujie further teaches wherein the processor is further configured to: based on detecting, in a portion of the first image data (Fig. 2 (E), the white portions of the image based on the three dimensional positions of the markers, [0042]), at least one of the first plurality of markers and not a marker of the second plurality of markers (Fig. 2 (E), the white portions show the portion of the surgical instruments, S2 of fig. 2 (A), not including tips, S1 of fig. 2 (A), of the surgical instruments; [0041] well-known methods may be used in which predetermined colors are applied to the main body parts S2 and the pieces of image information on the operating field image V1 are distinguished on the basis of the colors to identify the cutoff regions; wherein the main body parts S2 are considered as the shaft and marked by the markers as disclosed and modified Yates as mentioned above, Yates: 1312 of fig. 9 and [0060]), replace the portion of the first image data with a portion of the second image data ([0028] and [0042] in the composite image generating means 28, a composite image is generated by performing the following mask process. That is, first, as shown in FIG. 2 (E), a mask is generated by extracting the cutoff regions identified in the operating field image V1. Then, ranges of the sets of in-screen coordinates in the transformed image V3 (the drawing (D)) that match ranges of the in-screen coordinates of the cutoff regions in the operating field image V1 are identified as corresponding regions (dotted-lined regions in the drawing (D)) by the generated mask, and the pieces of image information on these corresponding regions are extracted. The pieces of image information on the cutoff regions in the operating field image V1 are thereafter superimposed or replaced with the pieces of image information on the corresponding regions, and the composite image shown in the drawing (F) is thereby generated) . 07-21-aia AIA Claim (s) 29 and 39 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Bailey et al. (US 20170272732 A1) Regarding claims 29 and 39, Fujie and Ren teach the surgical system of claim 21 and method of claim 31, Fujie and Ren do not teach wherein the surgical system comprises a display, and wherein the processor is further configured to: shift a first wavelength of the one or more wavelengths associated with the first image to a second wavelength, and wherein the second image is displayed on the display using the second wavelength. Bailey teaches wherein the surgical system (fig. 2) comprises a display (210 of figs. 2 and 3, [0028] a display screen), and wherein the processor (205 of fig. 2) is further configured to: shift a first wavelength of the one or more wavelengths associated with the first image to a second wavelength ([0018] and [0019] shift the wavelengths), and wherein the second image is displayed on the display using the second wavelength ([0037]). Taking the teachings of Fujie, Ren, and Bailey together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide the controllable wavelength ranges corresponding to the left eye and right eye frames, which may improve the extinction ratio of the glasses, and improving the extinction ratio may improve the 3D effects and reduce the likelihood of motion sickness . 07-21-aia AIA Claim (s) 30 and 40 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fujie et al. (US 20150145953 A1) in view of Ren et al. (US 20150173644 A1) as applied to claim 21, and further in view of Kessman et al. (US 6379302 B1) . Regarding claims 30 and 40, Fujie and Ren teach the surgical system of claim 21 and method of claim 31, Fujie and Ren do not teach wherein the processor is further configured to generate a third image based on the first image data or the second image data, and wherein the instructions cause the second image to be displayed side by side with the third image. Kessman teaches wherein the processor is further configured to generate a third image based on the first image data or the second image data, and wherein the instructions cause the second image to be displayed side by side with the third image (Col. 8, lines 47-60). Taking the teachings of Fujie, Ren, and Schoepp together as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the side by side display of the images of Kessman to into the display device of Fujie and Ren to provide more information and better image viewed by surgeon during surgery operation . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Ren et al. (US 20160183779 A1) discloses a method of visualizing an ophthalmic procedure can include receiving images of a surgical field acquired by an imaging device; identifying an area of interest; generating a graphical overlay including a magnified image of the area of the interest; and outputting the graphical overlay to a display device such that the graphical overlay is positioned over a field of view of a surgical microscope. Lee et al. (US 20130141654 A1) discloses the display panel includes a retardation film having a phase difference of about .lamda./4. The retardation film is attached on the side of the display panel from which light exits. The retardation film causes a phase shift of .lamda./4 to the light having a specific wavelength, .lamda.. Because the left eye polarizer in the polarizing glasses is orthogonal to the right eye polarizer, when the polarized right-eye image is displayed on the display panel, the right-eye image does not pass through the left-eye polarizer to the observer's left eye. Similarly, the observer's right eye cannot see the left-eye image when it is displayed on the display panel. Nagasawa et al. (US 20100275238 A1) discloses the synchronous rotating member 13 rotating in synchronization with the timing of field display of the TV is made of optical filters that allow passage of only particular wavelengths of RGB; the half of the disk is used for the left eye, and the remaining half is used for the right eye; the wavelength of the light for the right eye and the wavelength of the light for the left eye are changed by shifting the wavelength of each of RGB; and the eyeglasses 7 are made of optical filters that respectively allow passage of only the wavelength for the right eye and the wavelength for the left eye. Contact Information Any inquiry concerning this communication or earlier communications from the examiner should be directed to TUNG T VO whose telephone number is (571)272-7340. The examiner can normally be reached Monday-Friday 6:30 AM - 5:00 PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Pendleton can be reached at 571-272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. TUNG T. VO Primary Examiner Art Unit 2425 /TUNG T VO/Primary Examiner, Art Unit 2425 Application/Control Number: 19/211,913 Page 2 Art Unit: 2425 Application/Control Number: 19/211,913 Page 3 Art Unit: 2425 Application/Control Number: 19/211,913 Page 4 Art Unit: 2425 Application/Control Number: 19/211,913 Page 5 Art Unit: 2425 Application/Control Number: 19/211,913 Page 6 Art Unit: 2425 Application/Control Number: 19/211,913 Page 7 Art Unit: 2425 Application/Control Number: 19/211,913 Page 8 Art Unit: 2425 Application/Control Number: 19/211,913 Page 9 Art Unit: 2425 Application/Control Number: 19/211,913 Page 10 Art Unit: 2425 Application/Control Number: 19/211,913 Page 11 Art Unit: 2425 Application/Control Number: 19/211,913 Page 12 Art Unit: 2425 Application/Control Number: 19/211,913 Page 13 Art Unit: 2425 Application/Control Number: 19/211,913 Page 14 Art Unit: 2425 Application/Control Number: 19/211,913 Page 15 Art Unit: 2425 Application/Control Number: 19/211,913 Page 16 Art Unit: 2425
Prosecution Insights