DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. JP2023-160040, filed on September 25, 2023.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-11, 14-19, 22, 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Uyama (US20210244260A1) in view of Gunji (US20200311920A1).
Regarding claim 1, Uyama teaches a three-dimensional data generation system configured to generate three-dimensional data indicating a three-dimensional shape inside an object, the three-dimensional data generation system comprising: “a three-dimensional information generation unit that generates three-dimensional information” (Uyama, 0006).
an imaging apparatus that includes a tubular insertion unit configured to acquire an optical image inside the object and is configured to generate two or more images of the object in a state in which the insertion unit is inserted inside the object; “FIG. 1 illustrates a scene in which a surgeon (doctor) 5061 is during surgery on a patient 5071 on a patient bed 5069 using the endoscopic surgery system 5000. An endoscope operator 5062 holds an endoscope 5001 and inserts the endoscope 5001 into the body cavity of the patient” (Uyama, 0032). “Through the trocars 5025a to 5025d, a lens barrel 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient” (Uyama, 0033). “The imaging device 42a includes one imaging element 44a, and the three-dimensional information generation unit 14 generates the three-dimensional map D(X, Y, Z) (three-dimensional information) of the surgical field based on at least two surgical field images K(x, y) captured by the imaging device 42a at different times” (Uyama, 0110).
and a processor configured to:
acquire the two or more images; “The map generation unit 15 generates a three-dimensional map (hereinafter, simply referred to as a map) indicating a three-dimensional position of the surgical field and a three-dimensional position of the region-of-interest described below” (Uyama, 0047) and “FIG. 3 illustrates a scene in which an object 100, which is a stationary object, is observed by the imaging device 42a in a three-dimensional space XYZ having a point in the space as a reference position O. This scene assumes a case where the imaging device 42a captures a surgical field image K(x, y, t) at a predetermined timing, time t, for example, and captures a surgical field image K(x, y, t+Δt) at a timing different from the predetermined timing” (Uyama, 0057).
determine whether the two or more images include a graphics region in which the graphics information is superimposed; “the region-of-interest setting unit 20 sets a region-of-interest by performing a display of a region-of-interest frame indicating a region-of-interest so as to be superimposed on a surgical field image” (Uyama, 0065) and “the two imaging elements 44c and 44d are arranged in a state of maintaining a predetermined relative relationship and capture images of different locations of an affected part so as to partially overlap each other” (Uyama, 0139). By superimposing, it is interpreted as already determine.
execute first processing when it is determined that the two or more images do not include the graphics region, wherein the first processing includes generation processing of generating the three-dimensional data by using the two or more images; “FIG. 6 is an image illustrating an example in which a feature point extraction region has been set. As illustrated in FIG. 6, the above-described map generation unit 15 sets a mask 120 at the periphery of the screen, avoiding the central portion of the screen on which the region-of-interest frame 110 has been set. The map generation unit 15 extracts a feature point only inside the set mask 120. The region of the mask 120 that has been set is distant from the region-of-interest frame 110 indicating the position of the region-of-interest, and thus is not likely to have a large deformation during an operation. Accordingly, it is possible, at inside portions of the mask 120, to stably detect feature points regardless of the passage of time” (Uyama, 0082). It is interpreted that the 3d generation based on a mask excluding only areas with too strong deformation. “Based on the principle of three-dimensional surveying, the map generation unit 15 calculates three-dimensional coordinates (X.sub.A, Y.sub.A, Z.sub.A) of a point A in space using two-dimensional coordinates of the feature point A1 on the surgical field image K(x, y, t+Δt) and the two-dimensional coordinates of the feature point A2 on the surgical field image K(x, y, t+Δt)” (Uyama, 0060). “The three-dimensional information generation unit 14 generates the three-dimensional map D(X, Y, Z) (three-dimensional information) of the surgical field based on at least two surgical field images K(x, y)” (Uyama, 0110).
and execute second processing when it is determined that at least one image of the two or more images includes the graphics region, wherein the second processing includes processing of preventing the graphics region in the at least one image from contributing to generation of the three-dimensional data. “There is a case where images of objects unrelated to the surgical field, such as a surgical instrument including forceps 5023 or operator's fingers, are captured inside the mask 120 in the surgical field image K(x, y)” (Uyama, 0083). “The map generation unit 15 may have a function of removing objects such as surgical instruments and fingers registered in advance from the surgical field image K(x, y). This removal function is, for example, a function of performing image recognition for a pre-registered object and excluding the region in which the recognized object exists, from calculation” (Uyama, 0083). “According to the second embodiment, the display control unit 40 controls to display the surgical field image K(x, y) and the magnified surgical field image L(x, y) on one display device 50, with the two images adjacent to each other” (Uyama, 0119). “Furthermore, as illustrated in FIG. 2, the display device 50 may include a second display region 52b in which the surgical field image is to be displayed” (Uyama, 0054).
[0189] teaches a pair of imaging elements for acquiring image signals individually for the right eye and the left eye corresponding to stereoscopic vision (3D display) [0189], which the image in [0083] is at least one image of the two images. [0083] teaches a case where images of objects unrelated to the surgical field, such as surgical instrument including forceps 5023, are captured inside the mask 120 in the surgical field image, which this is determining that at least one image (surgical field image) of the two images includes the graphics region (surgical instrument including forceps 5023). [0083] teaches a function of removing objects such as surgical instruments in advance from the surgical field image. This removal function is a function of performing image recognition for a pre-registered object and excluding the region in which the recognized object exists, from calculation, which this is the second processing that includes processing of preventing the graphics region (surgical instrument) in the at least one image (surgical field image) from contributing to generation of the 3D data (the generated data output to 3D stereoscopic display).
Uyama does not teach the remaining claimed limitations,
However, Gunji teaches the below claimed limitations.
an image-processing device configured to superimpose graphics information on at least one image of the two or more images; “The image processing apparatus 300 may store, in the storage device 312, the information about the result of identification of the affected region 102, information about the size of the affected region 102, and the image data of a superimposed image on which the information is superimposed” (Gunji, 0102).
Uyama and Gunji are combinable because they are in the same field of endeavor of image processing. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine the system of Uyama with graphics information of Gunji in order preliminarily adjust the digital gain and gamma curve to set in order to be able to generate image data having high visibility for the surgeon 5061 and the endoscope operator 5062 (Uyama).
Regarding claim 2, the three-dimensional data generation system according to claim 1, Uyama teaches wherein the second processing includes processing of setting the graphics region in the at least one image as an ineffective region. “Avoiding the central portion of the screen on which the region-of-interest frame 110 has been set” (Uyama, 0082) and “when extracting a feature point from the surgical field image K(x, y) to which the region-of-interest has been set, it is desirable to extract the feature point from within the regions excluding the neighboring region of the region-of-interest” (Uyama, 0081).
Regarding claim 3, the three-dimensional data generation system according to claim 2, wherein the second processing includes the generation processing in which a region other than the ineffective region in the two or more images is used. “The map generation unit 15 may have a function of removing objects such as surgical instruments and fingers registered in advance from the surgical field image K(x, y). This removal function is, for example, a function of performing image recognition for a pre-registered object and excluding the region in which the recognized object exists, from calculation” (Uyama, 0083). It is interpreted that if there is an ineffective region, then everywhere else is effective.
Regarding claim 4, The three-dimensional data generation system according to claim 2, Uyama teaches:
wherein the second processing includes processing of setting the graphics region as the ineffective region based on setting information and position information, “when extracting a feature point from the surgical field image K(x, y) to which the region-of-interest has been set, it is desirable to extract the feature point from within the regions excluding the neighboring region of the region-of-interest” (Uyama, 0081).
wherein the setting information indicates an item corresponding to the graphics information superimposed on the at least one image, “it is assumed that feature points A1, B1, C1, D1, E1, F1, and H1, which are the vertices of the object 100, have been detected from within the surgical field image K(x, y, t)” (Uyama, 0058) and “the region-of-interest setting unit 20 sets a region-of-interest by performing a display of a region-of-interest frame indicating a region-of-interest so as to be superimposed on a surgical field image and designating the size, shape, and position of the region-of-interest frame” (Uyama, 0065).
and wherein the position information indicates a position of the graphics region in which the graphics information corresponding to the item is superimposed. “Subsequently, the region-of-interest setting unit 20 sets the region-of-interest frame 110 at the position of the input region-of-interest instruction information 105. The region-of-interest setting unit 20 controls to display the set region-of-interest frame 110 so as to be superimposed on the surgical field image K(x, y), as illustrated in FIG. 5B” (Uyama, 0071).
Regarding claim 5, The three-dimensional data generation system according to claim 2, Uyama teaches:
wherein the second processing includes processing of setting the graphics region as the ineffective region based on type information, setting information, and position information, “A region-of-interest is set by the operation of the region-of-interest setting unit 20. Specifically, the region-of-interest setting unit 20 sets a region-of-interest by performing a display of a region-of-interest frame indicating a region-of-interest so as to be superimposed on a surgical field image and designating the size, shape, and position of the region-of-interest frame” (Uyama, 0065). “FIG. 4 is a diagram illustrating an example of a method of setting the region-of-interest frame. FIG. 4A is a view illustrating an example of a surgical field image K(x, y) observed by the endoscope 5001. Hereinafter, information regarding the timing (for example, time) of capturing the surgical field image will be omitted” (0066). “the display device 50 may display various types of surgical information such as physical information of the patient and information regarding the surgical procedure” (0212).
wherein the type information indicates type of imaging apparatus, “Furthermore, the user inputs, through the input device, an instruction to change imaging conditions (type of irradiation light, magnification, focal length, or the like)” (0040).
wherein the setting information is prepared for each type of imaging apparatus and indicates an item corresponding to the graphics information superimposed on the at least one image, “region-of-interest setting unit 20 sets a region-of-interest by performing a display of a region-of-interest frame (interpreted as graphics information) indicating a region-of-interest (interpreted as item) so as to be superimposed on a surgical field image” (0065).
and wherein the position information indicates a position of the graphics region in which the graphics information corresponding to the item is superimposed. “Subsequently, the region-of-interest setting unit 20 sets the region-of-interest frame 110 at the position of the input region-of-interest instruction information 105. The region-of-interest setting unit 20 controls to display the set region-of-interest frame 110 so as to be superimposed on the surgical field image K(x, y), as illustrated in FIG. 5B” (0071).
Regarding claim 6, the three-dimensional data generation system according to claim 5, Uyama teaches wherein the processor is configured to receive the type information and the setting information input through an input device. “a setting signal instructing the setting of the region-of-interest is generated. On condition that the setting signal has been input, the region-of-interest setting unit 20 controls to display the region-of-interest frame 110” (0068). It is obvious to say that the processor receives type and setting information from an input device for the input device sends these details so the processor knows how to handle the connected hardware. The setting information is tailored to each type to indicate items corresponding to superimposed graphics on the image.
Regarding claim 7, The three-dimensional data generation system according to claim 2, Uyama teaches wherein the generation processing includes detection processing of detecting a feature point from the two or more images, “The map generation unit 15 first detects a feature point, which is a characteristic pixel, from within the surgical field image K(x, y, t) and the surgical field image K(x, y, t+Δt)” (Uyama, 0058).
and wherein the second processing includes:
processing of changing an image in the ineffective region such that a feature point is not detected from the ineffective region; “The region-of-interest setting unit 20 designate a region-of-interest, such as a tumor to be removed by surgery, from within the surgical field image captured by the imaging element 44a and converted into a visible image by the development processing unit” (0049). “When extracting a feature point from the surgical field image K(x, y) to which the region-of-interest has been set, it is desirable to extract the feature point from within the regions excluding the neighboring region of the region-of-interest” and “FIG. 6 is an image illustrating an example in which a feature point extraction region has been set. As illustrated in FIG. 6, the above-described map generation unit 15 sets a mask 120 at the periphery of the screen, avoiding the central portion of the screen on which the region-of-interest frame 110 has been set. The map generation unit 15 extracts a feature point only inside the set mask” (0082). It is interpreted that setting mask in a selected region of the image and performing feature point extraction only within that masked region while avoiding the region around the region-of-interest frame. feature points are not extracted from regions outside the mask including the region around region-of-interest. It excludes the neighborhood of the region-of-interest from feature point extraction by restricting the extraction area with a mask.
and the generation processing in which the two or more images are used. “The three-dimensional information generation unit 14 generates the three-dimensional map D(X, Y, Z) (three-dimensional information) of the surgical field based on at least two surgical field images K(x, y)” (0110).
Regarding claim 8, the three-dimensional data generation system according to claim 3, Uyama teaches wherein the generation processing includes detection processing of detecting a feature point from the two or more images, “The map generation unit 15 first detects a feature point, which is a characteristic pixel, from within the surgical field image K(x, y, t) and the surgical field image K(x, y, t+Δt)” (0058)
and wherein the second processing includes processing of excluding the ineffective region from a region in which the detection processing is executed. “When extracting a feature point from the surgical field image K(x, y) to which the region-of-interest has been set, it is desirable to extract the feature point from within the regions excluding the neighboring region of the region-of-interest”, “FIG. 6 is an image illustrating an example in which a feature point extraction region has been set. As illustrated in FIG. 6, the above-described map generation unit 15 sets a mask 120 at the periphery of the screen, avoiding the central portion of the screen on which the region-of-interest frame 110 has been set”, and “The map generation unit 15 extracts a feature point only inside the set mask 120. The region of the mask 120 that has been set is distant from the region-of-interest frame 110 indicating the position of the region-of-interest, and thus is not likely to have a large deformation during an operation. Accordingly, it is possible, at inside portions of the mask 120, to stably detect feature points regardless of the passage of time. Since the feature points can be extracted stably, it is possible to enhance the stability of accuracy” (0082).
Regarding claim 9, The three-dimensional data generation system according to claim 3, Uyama teaches wherein the generation processing includes detection processing of detecting a feature point from the two or more images, “it is desirable to extract the feature points after removing these objects” (0083)
and wherein the second processing includes processing of deleting a feature point detected from the ineffective region. “FIG. 6 is an image illustrating an example in which a feature point extraction region has been set” (0082).
Regarding claim 10, the three-dimensional data generation system according to claim 1, Uyama teaches wherein the second processing includes processing of canceling execution of the generation processing. “FIG. 10D illustrates an example in which the zoom processing unit 26 stops the generation of the magnified surgical field image L(x, y)” (0128).
Regarding claim 11, the three-dimensional data generation system according to claim 1, Uyama teaches wherein a storage medium stores two or more images, generated by the imaging apparatus, on which the graphics information is not superimposed by the image-processing device, “the imaging device 42c includes the two imaging elements 44c and 44d that image different ranges partially overlapping each other, and the three-dimensional information generation unit 14 generates three-dimensional information of the surgical field based on the two surgical field images K(x, y) captured by the two imaging elements 44c and 44d at the same time” (Uyama, 0142) and “the region-of-interest setting unit 20 controls to display region-of-interest instruction information 105 indicating the designated region so as to be superimposed on the surgical field image K(x, y)” (Uyama, 0070).
Regarding claim 14, The three-dimensional data generation system according to claim 1,
Uyama and the combination with Gunji teach wherein the graphics information includes at least one of type of optical adaptor attached to a distal end of the insertion unit, brightness of an image generated by the imaging apparatus, a date, a mark, and a magnification of an image generated by the imaging apparatus. “The object ID of the data set that is selected on the data selection screen 1601 in FIG. 16, the name 1702 of the object, the time and date 1703 of photography are displayed on a data view screen 1701” (Gunji, 0197)”. It would have been obvious to a person in the art to say that the brightness of an image is mandatory for an image when generated, and to modify visual information to include a date, a mark, and a magnification of an image as well as title and time generated by the imaging apparatus since that would allow user to have more information related to the image.
Regarding claim 15, the three-dimensional data generation system according to claim 1, Uyama teaches wherein the processor is configured to acquire the two or more images from a storage medium after the two or more images are stored on the storage medium. “The three-dimensional map data storage unit 24 stores the three-dimensional map of the surgical field generated by the map generation unit 15 described above” (0051).
Regarding claim 16, the three-dimensional data generation system according to claim 1, Uyama teaches wherein the insertion unit includes a lens and an image sensor. “The endoscope 5001 includes: a lens barrel 5003” (0036) and “the imaging device 42a includes an imaging element 44a. The imaging element 44a is represented by an imaging element (photoelectric conversion element) such as a complementary metal oxide semiconductor (CMOS) image sensor” (0045).
Regarding claim 17, The three-dimensional data generation system according to claim 16, Uyama teaches wherein the lens and the image sensor are built in a distal end of the insertion unit, “The endoscope 5001 includes: a lens barrel 5003 (also referred to as a scope), a region of a predetermined length from a distal end” (0036)
wherein positions of the distal end when the two or more images are generated are different from each other, “With the swivel movement of the configuration on the distal end side including the microscope unit 5303 about the second axis O.sub.2 and the third axis O.sub.3, it is possible to move the microscope unit 5303 so as to change the position” (0195) and “the region-of-interest setting unit 20 (setting unit) designates a specific position of the surgical field image K(x, y)” (0108).
and wherein orientations of the distal end when the two or more images are generated are different from each other. “This would make it possible to measure the position and orientation of the endoscope 5001 without capturing two images at different times by the imaging device 42a, enabling estimation of the position of the region-of-interest” (0113).
Regarding claim 18, the three-dimensional data generation system according to claim 1, Uyama teaches wherein the imaging apparatus, the image-processing device, and the processor are included in an endoscope device. “FIG. 13 is a diagram illustrating an example of a schematic configuration of a medical observation system in which an imaging device includes two imaging elements, and a camera control unit includes a tracking processing unit” (0019) and “The medical observation system 10a includes an imaging device 42a and a camera control unit 12a. Mounted on the camera head 5005 of the endoscope 5001 described above” (0044).
Regarding claim 19, The three-dimensional data generation system according to claim 1, wherein the imaging apparatus and the image-processing device are included in an endoscope device, and wherein the processor is included in an external device that is separate from the endoscope device. “The control device 5317 is provided as a separate device from the microscope device 5301” (0211).
Regarding claim 22, the claim is directed to a method claim with the similar limitations as claim 1 As such, claim 22 is rejected on the same grounds
Regarding claim 23, the claim is directed to a method claim with the similar limitations as claim 1 As such, claim 23 is rejected on the same grounds.
Claim(s) 12, 13, 20, 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Uyama in view of Gunji (US20200311920A1), and further in view of Yonishi (US11012624B2).
Regarding claim 12, the three-dimensional data generation system according to claim 1, Uyama does not teach the claimed limitation, but Yonishi teaches wherein the processor is configured to determine whether the two or more images include the graphics region based on setting information indicating whether the graphics information is superimposed on each of the two or more images. “The overlapped area may be superimposed on the image” (Yonishi, Col. 12, line 67) and “first setting information including information related to a first specified area in an image” (Yonishi, Col. 2, line 2).
Uyama, Gunji, and Yonishi are combinable because they are in the same field of endeavor of image processing. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine the system of Uyama and Gunji with graphics information of Yonishi in order to preliminarily adjust the digital gain and gamma curve to set in order to improve convenience of the function setting.
Regarding claim 13, the three-dimensional data generation system according to claim 1, Uyama does not teach the claimed limitation, but Yonishi teaches wherein the processor is configured to:
perform, on the two or more images, image processing of detecting the graphics region; “where the image analysis processing is the intrusion detection processing and the overlapped area exists in the specified area” (Yonishi, Col. 11, line 40)
and determine whether the two or more images include the graphics region based on a result of the image processing. “Obtains information related to the specified area that is specified on the image” (Yonishi, Col. 7, line 7).
Uyama, Gunji, and Yonishi are combinable because they are in the same field of endeavor of image processing. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine the system of Uyama and Gunji with graphics information of Yonishi in order to preliminarily adjust the digital gain and gamma curve to set in order to improve convenience of the function setting.
Regarding claim 20, the three-dimensional data generation system according to claim 1, Uyama in combination with Yonishi teaches wherein the imaging apparatus is configured to generate the two or more images based on an optical image formed through a monocular optical adaptor attached to a distal end of the insertion unit. “A monocular camera” (Uyama, 0110). “The imaging unit 101 also performs processing for converting an optical image focused on an imaging surface into digital electric signals by photoelectric conversion. The image processing unit 102 performs predetermined image processing with respect to the signal after the photoelectric conversion is performed in the imaging unit 101” (Yonishi, Col. 4, line 21). “Even in a case where camera shake occurs in the surgical field image K(x, y) captured by the endoscope 5001, it is possible to obtain a surgical field image K(x, y) and a magnified surgical field image L(x, y)” (Uyama, 0182).
Uyama, Gunji, and Yonishi are combinable because they are in the same field of endeavor of image processing. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine the system of Uyama and Gunji with graphics information of Yonishi in order to preliminarily adjust the digital gain and gamma curve to set in order to improve convenience of the function setting.
Regarding claim 21, the three-dimensional data generation system according to claim 1, Uyama in combination with Yonishi teaches wherein the same region of a component of the object is seen in at least two images included in the two or more images. “The above-described specified area is equivalent to a monitoring area for a moving object” (Yonishi, Col. 5, line 18) and “includes the two imaging elements 44c and 44d that image different ranges partially overlapping each other” (Uyama, 0142).
Uyama, Gunji, and Yonishi are combinable because they are in the same field of endeavor of image processing. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine the system of Uyama and Gunji with graphics information of Yonishi in order to preliminarily adjust the digital gain and gamma curve to set in order to improve convenience of the function setting.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIANA SZE whose telephone number is (571)272-9916. The examiner can normally be reached Monday-Thursday 6am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.S./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
