DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Claims 2, 14, and 19, along with generic claims 1, 9-13, 15, 18, and 20 in the reply filed on July 9th, 2026 is acknowledged. Claims 3-8 and 16-17 are withdrawn.
The Examiner acknowledges the correction of Claim dependencies regarding Claims 14-17 and 19-20 in the Preliminary Amendment, with no new matter entered.
Claim Objections
Claims 1, 13, and 18 are objected to because of the following informalities:
Claims 1, 13, and 18 recite “wherein the condition includes an operation of the endoscope system that causes the group of images has a missing portion” should read “wherein the condition includes an operation of the endoscope system that causes the group of images to have a missing portion”, in the case that the group of images having a missing portion is attributed to an operation of the endoscope system. If this is not the intended interpretation, then the claim language should be clarified to convey the appropriate interpretation. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 12 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 12 recites the limitation "the matching image" in line 2. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 13, 15 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kyperountas (US 20220071711 A1), in view of Kuriyama (US 20200084380 A1).
Regarding Claim 1, Kyperountas teaches an image processing device for use with an endoscope system (Abstract: “At least one example embodiment is directed to a device including a memory including instructions, and a processor that executes the instructions to generate, during a medical procedure being performed by a clinician on an internal region of a patient, image data and depth data for the internal region”; Paragraph [0041]: “The camera(s) 136 includes hardware and/or software for enabling collection of video, images, and/or depth information of a medical procedure. In at least one example embodiment, the camera 136 captures video and/or still images of a medical procedure being performed on a body of patient. As is known in endoscopy, arthroscopy, and the like, the camera 136 may be designed to enter a body and take real-time video of the procedure to assist the clinician with performing the procedure and/or making diagnoses”; Paragraph [0022]: “In at least one example embodiment, the current position of the endoscope or other instrument having the depth and/or imaging cameras can be indicated on the 3D depth model in order to help the user navigate to the part of the anatomy that has not been imaged”; Paragraph [0053]: “In operation 308, the method 300 includes generating a depth model or depth map of the internal region based on the depth data. For example, the depth model is generated during the medical procedure using the depth data received by the processor 116 from one or more cameras 136. Any known method may be used to generate the depth model. In at least one example embodiment, the depth model is generated in response to a determination that a medical instrument 140 used for the medical procedure enters a general region of interest”), the image processing device comprising:
one or more processors comprising hardware (Abstract: “At least one example embodiment is directed to a device including a memory including instructions, and a processor that executes the instructions to generate, during a medical procedure being performed by a clinician on an internal region of a patient, image data and depth data for the internal region), wherein the one or more processors being configured to (Paragraph [0032]: “FIG. 1 illustrates a system 100 according to at least one example embodiment. The system 100 includes an output device 104, a robotic device 108, a memory 112, a processor 116, a database 120, a neural network 124, an input device 128, a microphone 132, camera(s) 136, and a medical instrument or tooling 140”; Paragraph [0051]: “FIG. 3 illustrates a method 300 according to at least one example embodiment. In general, the method 300 may be performed by one or more of the elements from FIG. 1. For example, the method 300 is performed by the processor 116 based on various inputs from other elements of the system 100. However, the method 300 may be performed by additional or alternative elements in the system 100, under control of the processor 116 or another element, for example, as would be recognized by one of ordinary skill in the art”; Paragraph [0053]: “In operation 308, the method 300 includes generating a depth model or depth map of the internal region based on the depth data. For example, the depth model is generated during the medical procedure using the depth data received by the processor 116 from one or more cameras 136. Any known method may be used to generate the depth model. In at least one example embodiment, the depth model is generated in response to a determination that a medical instrument 140 used for the medical procedure enters a general region of interest”):
create a three-dimensional model of a subject from a group of images captured by an endoscope in the endoscope system (Paragraph [0053]: “In operation 308, the method 300 includes generating a depth model or depth map of the internal region based on the depth data. For example, the depth model is generated during the medical procedure using the depth data received by the processor 116 from one or more cameras 136. Any known method may be used to generate the depth model. In at least one example embodiment, the depth model is generated in response to a determination that a medical instrument 140 used for the medical procedure enters a general region of interest”; Paragraph [0015]: “By graphing/building a 3D depth model, regions within the anatomy can be identified that have not been imaged in color by an imaging sensor. For example, discontinuities (e.g., missing data) in the 3D depth model itself can indicate that the region around the discontinuity has not been imaged by the color image sensor”);
detect a non-observed region in which an image from the group of images is not captured by the endoscope based on the three-dimensional model (Paragraph [0027]: “As noted above, example embodiments are able to identify the regions of an anatomy that have not been imaged, or examined. This is accomplished by creating the 3D (depth) model and identify discontinuities or missing depth information in the model (i.e., identify regions of the 3D model where no depth data exists). Optionally, based on the aforementioned discontinuities in the depth information, the system can infer what image/visual information should also be missing, for example, by producing visualizations on a display that show the regions with missing data. In the event that depth data is less sparse than image data, the resolution of the depth sensor is considered in order to determine if image data is really missing”);
acquire operation signals of the endoscope system (Paragraph [0028]: “In order to determine if a missed/unexamined region should be examined, the system may identify gaps within the region that the user is examining (e.g., occluded areas). Optionally, the system uses input from the user about the overall region of interest. The user input may be in the form of identifying regions of interest on a generic 3D depth model. In at least one example embodiment, the system uses a machine learning algorithm or deep neural networks to learn, over time and over multiple manually performed procedures, the regions that should be examined for each specific type of procedure. The system can then compare this information against the regions that are actually being examined by the user to determine if any region that has not been imaged/examined should or should not be examined”. Notes: The broadest reasonable interpretation of operation signals of the endoscope system are any received signals relevant to the operation of an endoscope system in which the image processing device is being used with. Hence, operation signals are inclusive of images captured by the endoscope and received by the image processing device. Images captured by the endoscope are considered through a machine learning algorithm that determines whether the images captured by the endoscope missing certain regions necessitates the detection of a missing region to reimage. Therefore, the images acquired via the endoscope for image processing in the image processing device are considered operation signals of the endoscope system).
Determine whether the operation signal matches a condition, wherein the condition includes an operation of the endoscope system that causes the group of images to have missing portion (Paragraph [0027]: “As noted above, example embodiments are able to identify the regions of an anatomy that have not been imaged, or examined. This is accomplished by creating the 3D (depth) model and identify discontinuities or missing depth information in the model (i.e., identify regions of the 3D model where no depth data exists). Optionally, based on the aforementioned discontinuities in the depth information, the system can infer what image/visual information should also be missing, for example, by producing visualizations on a display that show the regions with missing data. In the event that depth data is less sparse than image data, the resolution of the depth sensor is considered in order to determine if image data is really missing”; Paragraph [0028]: “In order to determine if a missed/unexamined region should be examined, the system may identify gaps within the region that the user is examining (e.g., occluded areas). Optionally, the system uses input from the user about the overall region of interest. The user input may be in the form of identifying regions of interest on a generic 3D depth model. In at least one example embodiment, the system uses a machine learning algorithm or deep neural networks to learn, over time and over multiple manually performed procedures, the regions that should be examined for each specific type of procedure. The system can then compare this information against the regions that are actually being examined by the user to determine if any region that has not been imaged/examined should or should not be examined”. Notes: The broadest reasonable interpretation of operation signals of the endoscope system are any received signals relevant to the operation of an endoscope system in which the image processing device is being used with. Hence, operation signals are inclusive of images captured by the endoscope and received by the image processing device. Images captured by the endoscope are considered through a machine learning algorithm that determines whether the images captured by the endoscope missing certain regions necessitates the detection of a missing region to reimage. Therefore, the images acquired via the endoscope for image processing in the image processing device are considered operation signals of the endoscope system. The condition matched by the operational signal (images) is whether the images align with the desired anatomy of interest; the operation of the endoscope system results in the capture of images that may have a missing portion, which is integral to the check of the condition being matched by the operational signal (images)); and
Responsive to determining the operation signal matches the condition, stop identifying the non-observed region as a region to re-examine or image (Paragraph [0028]: “In order to determine if a missed/unexamined region should be examined, the system may identify gaps within the region that the user is examining (e.g., occluded areas). Optionally, the system uses input from the user about the overall region of interest. The user input may be in the form of identifying regions of interest on a generic 3D depth model. In at least one example embodiment, the system uses a machine learning algorithm or deep neural networks to learn, over time and over multiple manually performed procedures, the regions that should be examined for each specific type of procedure. The system can then compare this information against the regions that are actually being examined by the user to determine if any region that has not been imaged/examined should or should not be examined”. Notes: The broadest reasonable interpretation of detecting a non-observed region is detecting a discontinuity in data pertaining to a region. Considering that the system makes a determination that a non-observed region should or should not be imaged/examined, the non-observed region is no longer a subject of interest regarding a matching of the condition, and hence after a decision is made to not image/examine the non-observed region, the non-observed region is no longer subject to condition checks with regards to an established anatomy of interest).
Kyperountas does not explicitly teach that the image processing device, responsive to determining the operation signal matches the condition, temporarily stops detecting the non-observed region.
However, Kuriyama teaches responsive to determining the operation signal matches the condition, temporarily stops detecting a region (Paragraph [0069]: “The blur correction non-required region detection section 330 performs a process for detecting the blur correction non-required region on the object image output from the pre-processing section 320. The process for detecting the blur correction non-required region will be described later in detail. The blur correction non-required region detection section 330 may detect the blur correction non-required region in each frame, or stop detecting after detecting once until a specific requirement is satisfied”. Notes: upon determining the blur correction non-required region in an image frame, it stops detecting until a specific requirement is satisfied, which reads on temporarily stopping detection responsive to determining the operation signal (image) matches the condition (matches the region of interest)).
Kyperountas and Kuriyama are considered analogous in the art with respect to image processing devices used with endoscopes. A common motivation in the art is to stop detecting regions of interest (observed or non-observed) after a conditional check determining whether the region of interest matches a condition; doing so avoids repeating the detection of the region of interest after a determination has been made in accordance with a condition. This is evident in Kuriyama.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the stopping of identifying the non-observed region as a region to re-examine or image responsive to determining an operation signal matches a condition of Kyperountas with the temporary stopping of detecting a region responsive to operation signals matching a condition of Kuriyama; Doing so would yield the predictable result of temporarily stopping the detection of a non-observed region responsive to an operation signal matching a condition, preventing continued detection of a non-observed region determined to be not of interest.
Claim 13, being similar in scope to Claim 1, is rejected under the same rationale.
Claim 18, being similar in scope to Claim 1, is rejected under the same rationale.
Regarding Claim 15, the image processing method according to Claim 13 is rejected over Kyperountas as modified.
Kyperountas as modified teaches the image processing method according to Claim 13, wherein: the condition includes an operation of the endoscope system in which the non-observed region detection is unnecessary (Kyperountas, Paragraph [0027]: “As noted above, example embodiments are able to identify the regions of an anatomy that have not been imaged, or examined. This is accomplished by creating the 3D (depth) model and identify discontinuities or missing depth information in the model (i.e., identify regions of the 3D model where no depth data exists). Optionally, based on the aforementioned discontinuities in the depth information, the system can infer what image/visual information should also be missing, for example, by producing visualizations on a display that show the regions with missing data. In the event that depth data is less sparse than image data, the resolution of the depth sensor is considered in order to determine if image data is really missing”. Notes: non-observed region detection being unnecessary is inherent to the operation as described in Kyperountas, as in a case where a non-observed region is not detected during an operation of the endoscope system, the non-region detection is inherently unnecessary)
Claims 2, 14 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kyperountas (US 20220071711 A1), in view of Kuriyama (US 20200084380 A1), and in further view of Shiratani (US 20190268538 A1), Iketani (US 20220414956 A1) and Cho (Recent Advances in Image-enhanced Endoscopy, 2011).
Regarding Claim 2, the image processing device according to Claim 1 is rejected over Kyperountas as modified.
Kyperountas as modified does not teach that the operation signals include a signal that indicates switching types of illumination light provided to the endoscope from a light source.
However, Shiratani teaches that operation signals include a signal that indicates switching types of illumination light provided to the endoscope from a light source (Paragraph [0061]: “When the control section 43 detects, based on the instruction from the scope switch 24, that the observation mode of the endoscope system 1 is set in the narrowband light observation mode, the control section 43 generates a system control signal for causing the G light and the BS light to be supplied as illumination light to the endoscope 2, and outputs the system control signal to the light source control section 34. When the control section 43 detects, based on the presumption result obtained by the operation state presumption section 43a, that the operation related to discrimination of the desired object or the polyp, which is a target of finding, found in the subject into which the endoscope 2 is inserted is being performed, the control section 43 generates a system control signal for causing the image in which the specular reflection region is corrected by the specular reflection region correction section 42c to be outputted as an observation image to the display apparatus 5, and outputs the system control signal to the selector 42b and the display control section 42d”).
Kyperountas as modified and Shiratani are considered analogous in the art with respect to operation signals of endoscopes in relation to image processing devices. While Kyperountas does not explicitly describe that the operation signals include a signal that indicates switching types of illumination light provided to the endoscope from a light source, an image processing device receiving operation signals for endoscopes regarding the lighting type is common in the art. A common motivation for having the illumination light type be an operation signal is to conditionally perform tasks based on the illumination light operation signal, as is evident in Shiratani.
Therefore, it would have been obvious to a person having ordinary skill in the art to combine the operation signals of Kyperountas as modified with the operation signals of Shiratani; Doing so would yield the predictable result of an image processing device related to an endoscope being able to conditionally perform tasks and determinations based on a broader pool of operation signals.
Kyperountas as modified does not teach that the condition includes the switching types of the illumination light.
However, Iketani teaches that switching types of illumination light is a condition for examining different regions of an observation target (Paragraph [0048]: “The endoscope system 10 includes a mono-emission mode, a multi-emission mode, and an incision line display mode, which are be switched by the mode selector switch 12f. The mono-emission mode is a mode in which illumination light having the same spectrum is continuously applied to illuminate an observation target. The multi-emission mode is a mode in which a plurality of illumination light beams having different spectra are applied while being switched therebetween according to a specific pattern to illuminate an observation target. The illumination light includes normal light (broadband light such as white light) or special light used for emphasizing a specific region of an observation target. In the mono-emission mode, switching to illumination light having another spectrum may be performed by operating the mode selector switch 12f. For example, first illumination light and second illumination light having different spectra may be switched”).
Kyperountas as modified and Iketani are considered analogous in the art with respect to image processing devices utilizing endoscopes with the ability to change illumination type. One would be motivated to image and examine different regions within a target operation area with different lighting, as is evident in Iketani, as different types of lighting can emphasize different regions more effectively.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the switching of illumination type of Kyperountas as modified with the switching of illumination type as a condition for viewing different regions of Iketani; Doing so would yield the predictable result of enabling effective imaging of particular regions.
Kyperountas as modified does not explicitly teach that illumination light type is a condition with respect to the detection of non-observed regions.
However, Cho teaches that illumination light type is a condition with respect to the detection of non-observed regions (NBI is a novel endoscopic technique that may enhance the accuracy of diagnosis using narrow-bandwidth filters in a red green-blue (RGB) sequential illumination system.1 This technique produces different images at distinct levels of the mucosa and increases the contrast between the epithelial surface and the subjacent vascular pattern. NBI may provide the same contrast-enhancement capability as chromoendoscopy with out requiring the use of dye agents.2 The basic principle of NBI is that the depth of penetration into the mucosa depends on the wavelength used: superficial for the blue band, deep for the red band, and intermediate for the green band.3 Because gastrointestinal cancers originate in the mucosa, the use of blue-colored, short-wavelength visible light, which can penetrate only into the mucosa, may be helpful in the observation of minute early expressions).
Kyperountas as modified and Cho are considered analogous in the art with respect to the use of endoscopes for imaging regions of interest. A common motivation in the art is to utilize different illumination types for detecting specific regions with endoscopes, as is evident in Cho.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the detection of particular regions through illumination type switching as a condition of Kyperountas as modified with the detection of non-observed regions through the use of specific illumination types; Doing so would yield the predictable result of the switching of illumination types serving as a condition for determining whether a non-observed region should be detected.
Claim 14, being similar in scope to Claim 2, is rejected under the same rationale.
Claim 19, being similar in scope to Claim 2, is rejected under the same rationale.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Kyperountas (US 20220071711 A1), in view of Kuriyama (US 20200084380 A1), and in further view of Leist (US 20210378748 A1).
Regarding Claim 9, the image processing device according to Claim 1 is rejected over Kyperountas as modified.
Kyperountas as modified teaches a user interface that accepts a user operation (Kuriyama, Paragraph [0073]: “The display section 400 is, for example, a liquid crystal monitor, and sequentially displays the combined images output from the image combining section 350. The external I/F section 500 is an interface used for input to the endoscope apparatus by the user or the like”).
Kyperountas as modified does not teach that the one or more processors are being configured to select at least one of a plurality of conditions based on the user operation accepted by the user interface and sets a selected condition to be the condition.
However, Leist teaches one or more processors are being configured to select at least one of a plurality of conditions based on the user operation accepted by the user interface and sets a selected condition to be the condition (Paragraph [0092]: “In certain embodiments, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices. In general, a processor”; Paragraph [0076]: “As an example, a user control may allow a surgical team member to adjust a maximum depth (e.g., MaxD in an algorithm described above) within which embedded anatomy is visually represented in an image and beyond which an embedded anatomy is not visually represented in an image. This may allow a surgical team member to control how deep beyond visible surface tissue that embedded anatomy will be visualized. As another example, a user control may allow a surgical team member to adjust a minimum visualization threshold (e.g., MinCE in an algorithm described above) for embedded anatomy. This may allow the surgical team member to control a minimum allowable visual emphasis (e.g., a minimum opacity) that is to be displayed for embedded anatomy. As another example, a user control may allow a surgical team member to adjust a prominence multiplier that adjusts a degree of visual emphasis that is used to visually represent embedded anatomy in an image. This may allow the surgical team member to control how prominently embedded anatomy is visually emphasized in an image”. Notes: multiple conditions, such as maximum depth, minimum visualization threshold, and prominence multiplayer are available, and selectable as a condition to be set, where depending on the set value, a non-observed region may or may not be detected).
Kyperountas as modified and Leist are considered analogous in the art with respect to image processing related to endoscope operation. A common motivation is to enable different conditions to be set and selected, as doing so would allow a broader set of regions for imaging/examination to be analyzed under specific conditions for doing so. Multiple different conditions result in the detection of non-observed regions, as is evident in Leist.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the setting of a condition with respect to detection of a non-observed region of Kyperountas as modified with the setting and selection of multiple conditions of Leist; Doing so would yield the predictable result of having a broader set of conditions related to the detection of non-observed regions.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Kyperountas (US 20220071711 A1), in view of Kuriyama (US 20200084380 A1), and in further view of Saito (US 20170100018 A1) and Odlivak (US 20050073578 A1).
Regarding Claim 10, the image processing device according to Claim 1 is rejected over Kyperountas as modified.
Kyperountas as modified teaches a storage (Kyperountas, Paragraph [0119]: “Aspects of the present disclosure may take the form of an embodiment that is entirely hardware, an embodiment that is entirely software (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module,” or “system.” Any combination of one or more computer-readable medium(s) may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium”).
Kyperountas as modified does not teach a storage, wherein the storage is configured to store a correspondence relationship between combinations of device models of a plurality of devices comprising the endoscope system and the condition, wherein the plurality of devices include the endoscope and a light source configured to provide illumination light to the endoscope, and the one or more processors being configured to acquire device model information of each of the plurality of devices to which the image processing device is connected.
However, Saito teaches a storage is configured to store a correspondence relationship between combinations of device models of a plurality of devices comprising the endoscope system (Paragraph [0037]: “The endoscope 2A includes a video interface (abbreviated as video I/F) 23a configured by a buffer circuit and the like connected to the image pickup device 22a, a nonvolatile memory 24a, and an endoscope-ID storing section (in FIG. 2, simply described as endoscope ID) 25a in which endoscope identification information (endoscope ID) including endoscope type information is stored. Note that, the endoscope type is the same meaning as an endoscope kind”), wherein the plurality of devices includes the endoscope and a light source configured to provide illumination light to the endoscope (Paragraph [0025]: “FIG. 15 is a diagram showing, in a table format, a part of setting data for setting the plurality of functions of the endoscope system to the appropriate setting states according to a type of a video processor configuring the endoscope system and a type of a light source apparatus”), and the one or more processors being configured to acquire device model information of each of the plurality of devices to which the image processing device is connected (Paragraph [0030]: “As shown in FIG. 1, an endoscope system 1 in a first embodiment of the present invention includes endoscopes 2A, 2B, . . . (in FIGS. 1 and 2, only 2A and 2B are shown) inserted into a body of a patient, a light source apparatus 3 that supplies illumination light to an endoscope 2I (I=A, B, . . . ) connected to the light source apparatus 3, a video processor 4 forming a signal processing apparatus that performs signal processing for an image pickup device mounted on the connected endoscope 2I, a monitor 5 functioning as a display apparatus that displays a video signal (an image signal) generated by the video processor 4, a keyboard 6 connected to the video processor 4 and forming an input section, a VTR 7 functioning as a recording apparatus, a printer 8, and a network apparatus 9. Note that, in the present embodiment, the light source apparatus 3, the monitor 5, the VTR 7, the printer 8, and the network apparatus 9 form a plurality of peripheral apparatuses connected to the video processor 4 forming the signal processing apparatus”; Paragraph [0037]: “an endoscope-ID storing section (in FIG. 2, simply described as endoscope ID) 25a in which endoscope identification information (endoscope ID) including endoscope type information is stored. Note that, the endoscope type is the same meaning as an endoscope kind”).
Kyperountas as modified and Saito are considered analogous in the art with respect to image processing systems with endoscopes. A common motivation in the art is to store specific information regarding endoscopes and illumination devices, as doing so enables setting parameters related to the two devices, as is evident in Saito.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the image processing device with an endoscope of Kyperountas with the storage of corresponding device models comprising the endoscope condition and light source of Saito; Doing so would yield the predictable result of the setting of parameters specific to the devices of interest.
Kyperountas as modified does not teach a correspondence relationship between combinations of device models of a plurality of devices comprising the endoscope system and the condition and does teach acquiring the condition corresponding to one of the combinations of the device models from the storage.
However, Odlivak teaches a correspondence relationship between combinations of device models of a plurality of devices comprising the endoscope system and the condition and acquiring the condition corresponding to one of the combinations of the device models from the storage (Paragraph [0049]: “As further shown in the image capture interface screen 200 of FIG. 11, a Scope Information section 225 is displayed providing functionality for enabling selection of the Scope Information, for example, when a type of scope is changed during a procedure. The scope information is further associated with the stored images and any corresponding report and patient record generated and stored within the EW system”. Notes: The condition matched by the operational signal (images) is whether the images align with the desired anatomy of interest; the operation of the endoscope system results in the capture of images that may have a missing portion, which is integral to the check of the condition being matched by the operational signal (images). Therefore, the storing of images taken by an endoscope in relation to the endoscope are considered the establishment of a corresponding relationship between the device and the condition).
Kyperountas as modified and Odlivak are considered analogous in the art with respect to storing endoscope information related to the processing of images captured by endoscopes. A common motivation is to organize information specific to the images taken for documentation purposes, as is evident in Odlivak.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the storage of device combinations of Kyperountas as modified with the storage of devices and conditions pertaining to operation signals of Odlivak; Doing so would yield the predictable result of organizing and establishing correspondences between device combinations and conditions pertaining to operation signals of the devices.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Kyperountas (US 20220071711 A1), in view of Kuriyama (US 20200084380 A1), and in further view of Stack Overflow (remove image if condition jquery, 2012).
Regarding Claim 11, the image processing device according to Claim 1 is rejected over Kyperountas as modified.
Kyperountas as modified does not explicitly teach removing a matching image that corresponds to the condition subsequent to temporarily stopping detecting the non-observed region.
However, Stack Overflow teaches removing a matching image corresponding to the condition subsequent to a condition (post by hyperrjas: “I just want to remove from ".preview" the images which have a width or height less than 50px”; answer by Rob W: “You have to loop through the individual img elements”).
Kyperountas as modified and Stack Overflow are analogous in the art with respect to the processing of obtained images. It is well known that within a collection of images, certain images can be removed based on a condition, as is evident in Stack Overflow. While Stack Overflow does not teach the removal of an image in accordance with a condition specific to operation signals of an endoscope, a person having ordinary skill in the art before the effective filing date of the claimed invention would have appreciated that the concept of doing so is well known within the realm of image processing; One would be motivated to do so to remove unnecessary stored data.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the temporary stopping of detecting the non-observed region of Kyperountas as modified with the subsequent removal of matching images based on a condition of Stack Overflow; Doing so would yield the predictable result of removing unnecessary stored data.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Kyperountas (US 20220071711 A1), in view of Kuriyama (US 20200084380 A1), Saito (US 20170100018 A1) and Odlivak (US 20050073578 A1), and in further view of Stack Overflow (remove image if condition jquery, 2012).
Regarding Claim 12, the image processing device according to Claim 10 is rejected over Kyperountas as modified.
Kyperountas as modified teaches one or more processors being configured to detect the non-observed regions (Kyperountas, Paragraph [0027]: “As noted above, example embodiments are able to identify the regions of an anatomy that have not been imaged, or examined. This is accomplished by creating the 3D (depth) model and identify discontinuities or missing depth information in the model (i.e., identify regions of the 3D model where no depth data exists). Optionally, based on the aforementioned discontinuities in the depth information, the system can infer what image/visual information should also be missing, for example, by producing visualizations on a display that show the regions with missing data. In the event that depth data is less sparse than image data, the resolution of the depth sensor is considered in order to determine if image data is really missing”; Kyperountas, Paragraph [0036]: “The processor 116 may correspond to one or many computer processing devices. For instance, the processor 116 may be provided as a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), any other type of Integrated Circuit (IC) chip, a collection of IC chips, a microcontroller, a collection of microcontrollers, or the like. As a more specific example, the processor 116 may be provided as a microprocessor, Central Processing Unit (CPU), and/or Graphics Processing Unit (GPU), or plurality of microprocessors that are configured to execute the instructions sets stored in memory 112. The processor 116 enables various functions of the system 100 upon executing the instructions stored in memory 112”)
Kyperountas as modified does not teach the subsequent removing of a matching image.
However, Stack Overflow teaches the removing of a matching image (post by hyperrjas: “I just want to remove from ".preview" the images which have a width or height less than 50px”; answer by Rob W: “You have to loop through the individual img elements”).
Kyperountas as modified and Stack Overflow are analogous in the art with respect to the processing of obtained images. It is well known that within a collection of images, certain images can be removed based on a condition, as is evident in Stack Overflow. While Stack Overflow does not teach the removal of an image in accordance with a condition specific to operation signals of an endoscope, a person having ordinary skill in the art before the effective filing date of the claimed invention would have appreciated that the concept of doing so is well known within the realm of image processing; One would be motivated to do so to remove unnecessary stored data.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the temporary stopping of detecting the non-observed region of Kyperountas as modified with the subsequent removal of matching images based on a condition of Stack Overflow; Doing so would yield the predictable result of removing unnecessary stored data.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Kyperountas (US 20220071711 A1), in view of Kuriyama (US 20200084380 A1), and in further view of Shiratani (US 20190268538 A1).
Regarding Claim 20, the computer-readable, non-transitory storage medium according to Claim 18 is rejected over Kyperountas as modified.
Kyperountas teaches the computer-readable, non-transitory storage medium according to Claim 18, wherein: the condition includes an operation of the endoscope system in which the non-observed region detection is unnecessary (Kyperountas, Paragraph [0027]: “As noted above, example embodiments are able to identify the regions of an anatomy that have not been imaged, or examined. This is accomplished by creating the 3D (depth) model and identify discontinuities or missing depth information in the model (i.e., identify regions of the 3D model where no depth data exists). Optionally, based on the aforementioned discontinuities in the depth information, the system can infer what image/visual information should also be missing, for example, by producing visualizations on a display that show the regions with missing data. In the event that depth data is less sparse than image data, the resolution of the depth sensor is considered in order to determine if image data is really missing”. Notes: non-observed region detection being unnecessary is inherent to the operation as described in Kyperountas, as in a case where a non-observed region is not detected during an operation of the endoscope system, the non-region detection is inherently unnecessary).
Kyperountas as modified does not teach that the condition includes changing the magnification of the endoscope.
However, Shiratani teaches changing the magnification of the endoscope (Paragraph [0026]: “More specifically, the scope switch 24 is provided with, for example, an observation mode setting switch (not shown) that can issue to the processor 4 an instruction for setting an observation mode of the endoscope system 1 to either a white light observation mode or a narrowband light observation mode. The scope switch 24 is provided with, for example, an optical zoom switch (not shown) that can issue to the processor 4 an instruction related to optical magnification”).
Kyperountas as modified and Shiratani are considered analogous in the art with respect to operation signals of endoscopes in relation to image processing devices. While Kyperountas does not explicitly describe that the operation signals include a signal that indicates changing the magnification of an endoscope, an image processing device receiving operation signals for endoscopes regarding the magnification or other endoscope parameters is common in the art. A common motivation for having the magnification be an operation signal is to conditionally perform tasks based on the magnification operation signal, as is evident in Shiratani.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the image processing device with an endoscope used for detecting non-observed regions of Kyperountas with the changing of the magnification of the endoscope of Shiratani; doing so would yield the predictable result of enabling the detection of non-observed regions based on the magnification level.
Kyperountas as modified does not explicitly teach that the change in magnification is a condition. However, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention that depending on the magnification of the endoscope, certain regions will not be visible; Hence, the changing the magnification is a condition for establishing whether a non-observed region should be ignored or not, depending on whether the magnification is the result of the non-observation of the region.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention that changing the magnification of an endoscope serves as a condition for establishing whether a non-observed region should be ignored or not.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAYMOND CHUN LAM LI whose telephone number is (571)272-5124. The examiner can normally be reached M-F 8:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAYMOND CHUN LAM LI/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights