DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-7, 10-20, 24-26 and 28 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Mitsuhashi (US 2017/0245736 A1).
Regarding claim 1: Mitsuhashi discloses an image processing component (Fig. 2, image processing unit 43) to receive image data (The image processing unit 43 performs predetermined image processing on the imaging signal input from the received signal processing unit 42, and generates an image to be displayed on the display unit 46. The image processing unit 43 includes a processor such as a CPU. The image processing unit 43 includes an image generation unit 431, a movement amount acquisition unit 432, an association unit 433, and a display image generation unit 434., par. 54) of a view provided by an endoscope (Fig. 2, capsule endoscope system 1) of an area to be addressed that is inside a body of a patient (The display unit 46 displays an in-vivo image or the like based on the image received from the capsule endoscope 2., par. 61); an image analysis engine to: receive a first image of the area to be addressed, the first image from the image data received (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). , par. 105) by the image processing component (The image processing unit 43 performs predetermined image processing on the imaging signal input from the received signal processing unit 42, and generates an image to be displayed on the display unit 46. The image processing unit 43 includes a processor such as a CPU. The image processing unit 43 includes an image generation unit 431, a movement amount acquisition unit 432, an association unit 433, and a display image generation unit 434., par. 54); analyzing the first image (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42)., par. 105); and generate additional data based on the analysis of the first image (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42)., par. 105); and a display engine to: receive a second image of the area to be addressed (When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44)., par. 105), the second image from the image data received by the image processing component (The image processing unit 43 performs predetermined image processing on the imaging signal input from the received signal processing unit 42, and generates an image to be displayed on the display unit 46. The image processing unit 43 includes a processor such as a CPU. The image processing unit 43 includes an image generation unit 431, a movement amount acquisition unit 432, an association unit 433, and a display image generation unit 434., par. 54); receive the additional data generated by the image analysis engine (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105); generate a combined image that combines the additional data with the second image of the area to be addressed (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105); and provide the combined image for presentation to a viewer (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Regarding claim 2: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein analyzing the first image (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). , par. 105) includes determining one or more characteristics of the first image, and wherein the additional data is generated using the one or more characteristics of the first image (The association unit 433 associates a first image captured by an imaging unit with a second image captured with an effective imaging area continued from an effective imaging area of the imaging unit upon capturing the first image, on the basis of at least an imaging distance of the imaging unit in the movement direction and the movement amount of the capsule endoscope 2 acquired by the movement amount acquisition unit 432. The association unit 433 associates the second image with the first image based on the imaging signal captured by the first imaging unit 21A. The second image is an image selected from an image group based on the imaging signals captured by the second imaging unit 21B, and captured with the effective imaging area continued from the effective imaging area of the first imaging unit 21A upon capturing the first image. For a candidate image captured by the second imaging unit 21B which is a candidate for the second image, the association unit 433 calculates a first distance along the movement direction between a position of the first imaging unit 21A upon capturing the first image and a position of the second imaging unit 21B upon capturing the candidate image, on the basis of the movement amount of the capsule endoscope 2 in the movement direction, acquired by the movement amount acquisition unit 432, compares the calculated first distance and the imaging distance with each other, and determines whether to associate the candidate image as the second image, with the first image. The imaging distance is a distance between both ends of an effective imaging area along the movement direction, in the effective imaging area in which the predetermined brightness and the predetermined resolving power is provided for an image captured by an imaging unit. In the capsule endoscope 2, the imaging distance is the sum of the observation depth F.sub.1 of the first imaging unit 21A and the observation depth F.sub.2 of the second imaging unit 21B., par. 57).
Regarding claim 3: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the image data comprises video, and wherein the first image (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). , par. 105) comprises a frame of the video (An image display device captures the images recorded in the recording medium, performs predetermined image processing on the images, and then plays back the images as in-vivo images while switching the images frame by frame (frame playback). A physician or the like as an operator diagnoses the in-vivo images played back on the image display device to diagnose the subject., par. 5).
Regarding claim 4: Mitsuhashi satisfies all the elements of claim 2. Mitsuhashi further discloses wherein the second image ( When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44)., par. 105) comprises another frame of the video (An image display device captures the images recorded in the recording medium, performs predetermined image processing on the images, and then plays back the images as in-vivo images while switching the images frame by frame (frame playback). A physician or the like as an operator diagnoses the in-vivo images played back on the image display device to diagnose the subject., par. 5).
Regarding claim 5: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the combined image (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105) is generated to be a frame of video, and wherein the display engine generates a plurality of combined frames comprising video (An image display device captures the images recorded in the recording medium, performs predetermined image processing on the images, and then plays back the images as in-vivo images while switching the images frame by frame (frame playback). A physician or the like as an operator diagnoses the in-vivo images played back on the image display device to diagnose the subject., par. 5).
Regarding claim 6: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses further comprising: a splitter to receive the image data as input and to provide one or more similar copies of the image data as output (The receiving device 4 includes a reception unit 41, a received signal processing unit 42, an image processing unit 43, a control unit 44, an operating unit 45, a display unit 46, a memory 47, a data transmission/reception unit 48, and a power supply unit 49 supplying power to these units., par. 51) (The processing device 5 includes for example a workstation including the display device 6 such as a liquid crystal display., par. 66).
Regarding claim 7: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the image data is received from one or more displays (Fig. 2, display unit 46 and display device 6) to which the image processing component (Fig. 2, image processing unit 43) is connected (Fig. 2).
Regarding claim 10: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses further comprising an image normalize engine to one or more of: normalize, adjust resolution, adjust color, adjust brightness, adjust contrast, adjust color space, invert image, rotate image, flip image, convert between analog and digital, partially process to prepare for further subroutines, and prepare for an internal format (The image generation unit 431 generates an image on the basis of the imaging signal input from the received signal processing unit 42. The image generation unit 431 performs, on the input imaging signal, optical black subtraction (0B) processing, demosaicing processing, density conversion (gamma conversion or the like) processing, smoothing (noise removal or the like) processing, white balance (WB) adjustment processing, synchronization processing, electronic zooming, edge enhancement processing, and the like corresponding to the type of the first imaging unit 21A, and outputs the generated image. The image generation unit 431 associates the generated image with information identifying whether any of the first imaging unit 21A and the second imaging unit 21B performs the imaging., par. 55).
Regarding claim 11: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein a plurality of combined images generated by the display engine form video of the endoscopic procedure (The receiving device sequentially receives the images transmitted from the capsule endoscope, and causes a recording medium to sequentially record the images. An image display device captures the images recorded in the recording medium, performs predetermined image processing on the images, and then plays back the images as in-vivo images while switching the images frame by frame (frame playback). A physician or the like as an operator diagnoses the in-vivo images played back on the image display device to diagnose the subject. , par. 5); (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Regarding claim 12: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the display engine uses a value representing color of one or more pixels in the image data to generate a contrasting color value is used for any additional data displayed near that location (The casing 29 is an outer casing formed in a size small enough to be introduced into an organ of the subject H, and is achieved by closing opening ends on both sides of a cylindrical casing 29C by domed casings 29A and 29B. The domed casings 29A and 29B are dome shaped optical members transparent to light of a predetermined wavelength band, such as visible light. The cylindrical casing 29C is a colored casing substantially opaque to visible light. As illustrated in FIG. 3, the casing 29 including such a cylindrical casing 29C and domed casings 29A and 29B liquid-tightly encapsulates the component parts of the capsule endoscope 2., par. 49).
Regarding claim 13: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the display engine provides combined output containing a mixture of image data and additional data (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105), wherein the combined output (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46). , par. 105) is provided to at least one of: a primary display (Fig. 2, display unit 46), a secondary display (Fig. 2, display device 6), a virtual display, and any other component of an endoscopic system used for one or more of the following: storage of the combined output and display of the combined output to a user (The image processing unit 43 performs predetermined image processing on the imaging signal input from the received signal processing unit 42, and generates an image to be displayed on the display unit 46., par. 54).
Regarding claim 14: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the display engine is to output a series of frames as video ( An image display device captures the images recorded in the recording medium, performs predetermined image processing on the images, and then plays back the images as in-vivo images while switching the images frame by frame (frame playback). A physician or the like as an operator diagnoses the in-vivo images played back on the image display device to diagnose the subject., par. 5), and the combined image is one of the series of combined frames (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Regarding claim 15: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the display engine can output one or more frames including only image data of a view provided by the endoscope, without any additional data from the image analysis engine (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Regarding claim 16: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the display engine can output one or more frames including only the additional data from the image analysis engine, without any image data provided by the endoscope (The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). , par. 105).
Regarding claim 17: Mitsuhashi satisfies all the elements of claim 16. Mitsuhashi further discloses further comprising a video mixer to combine the one or more frames output by the display engine with image data containing one or more frames from an endoscope (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Regarding claim 18: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein one or more of the image processing unit, the image analysis engine, the display engine, and a video mixer is contained in one or more of: a single component of an endoscopic system (Fig. 2, receiving device 4), more than one component of the endoscopic system, a component external to the endoscopic system, a virtual component, a cloud based component, and any combination thereof.
Regarding claim 19: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the functionality of the image analysis engine and the display engine are in one or more of: the same physical circuitry (Fig. 2, receiving device 4)and different physical circuitry.
Regarding claim 20: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the additional data generated by the image analysis engine (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105) is related to one or more of: general image category (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105), anatomic sublocation category, tissue category, procedure category, and another category pertinent to functionality of the endoscopic system.
Regarding claim 22: Mitsuhashi satisfies all the elements of claim 20. Mitsuhashi further discloses wherein the additional data generated by the image analysis engine includes image characteristics associated with a category associated with general image (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105) is related to one or more of: general image category (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Regarding claim 24: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the additional data generated by the image analysis engine is one or more of: provided to and obtained from a control engine (The control unit 23 controls operation processes of the component parts of the capsule endoscope 2. For example, when the first imaging unit 21A and the second imaging unit 21B alternately perform imaging processing, the control unit 23 controls the first imaging unit 21A and the second imaging unit 21B to alternately expose and read the imaging elements 21A-1 and 21B-1, and controls the first illumination unit 22A and the second illumination unit 22B to emit illumination light according to exposure timing of corresponding imaging units., par. 44); wherein the control engine is to control (The control unit 23 controls operation processes of the component parts of the capsule endoscope 2. For example, when the first imaging unit 21A and the second imaging unit 21B alternately perform imaging processing, the control unit 23 controls the first imaging unit 21A and the second imaging unit 21B to alternately expose and read the imaging elements 21A-1 and 21B-1, and controls the first illumination unit 22A and the second illumination unit 22B to emit illumination light according to exposure timing of corresponding imaging units. , par. 44) one or more endoscopic components of the endoscopic system (Fig. 2, capsule endoscopic system 1).
Regarding claim 25: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the additional data generated by the image analysis engine is used to identify one or more characteristics in the category of procedure related to a step of a procedure (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Regarding claim 26: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the additional data generated by the image analysis engine (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105) includes indication of a procedure-related item (Fig. 15, YES or NO determination).
Regarding claim 28: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses an image processing component (Fig. 2, image processing unit 43) to receive image data of a view (The image processing unit 43 performs predetermined image processing on the imaging signal input from the received signal processing unit 42, and generates an image to be displayed on the display unit 46. The image processing unit 43 includes a processor such as a CPU. The image processing unit 43 includes an image generation unit 431, a movement amount acquisition unit 432, an association unit 433, and a display image generation unit 434., par. 54) provided by an endoscope (Fig. 2, capsule endoscope system 1) of an area to be addressed that is inside a body of a patient (The display unit 46 displays an in-vivo image or the like based on the image received from the capsule endoscope 2., par. 61); an image analysis engine to: receive the image data of the area to be addressed (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). , par. 105) from the image processing component (The image processing unit 43 performs predetermined image processing on the imaging signal input from the received signal processing unit 42, and generates an image to be displayed on the display unit 46. The image processing unit 43 includes a processor such as a CPU. The image processing unit 43 includes an image generation unit 431, a movement amount acquisition unit 432, an association unit 433, and a display image generation unit 434., par. 54); analyze the image data (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42)., par. 105); and generate additional data based on the analysis of the image data (The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42)., par. 105); and a display engine to: receive the image data of the area to be addressed from the image processing component (When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44)., par. 105); receive the additional data generated by the image analysis engine (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105); generate output image frames comprising one or more of: the image data; and the additional data provide the output image frames for presentation to a viewer (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 8-9, 21, 23 and 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mitsuhashi in view of Wolf et al. (hereinafter Wolf) (US 2020/0273581 A1).
Regarding claim 8: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the image analysis engine (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105)
Mitsuhashi fails to specifically address uses an algorithm implemented in one or more of: an artificial intelligence routine, a neural network routine, a machine learning routine, a deep learning routine, and any routine that simulates human methodology.
Wolf discloses uses an algorithm implemented in one or more of: an artificial intelligence routine, a neural network routine, a machine learning routine, a deep learning routine, and any routine that simulates human methodology (In some embodiments, machine learning algorithms (also referred to as machine learning models in the present disclosure) may be trained using training examples, for example in the cases described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regressions algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbors algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recursive neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and so forth. For example, a trained machine learning algorithm may comprise an inference model, such as a predictive model, a classification model, a regression model, a clustering model, a segmentation model, an artificial neural network (such as a deep neural network, a convolutional neural network, a recursive neural network, etc.), a random forest, a support vector machine, and so forth. In some examples, the training examples may include example inputs together with the desired outputs corresponding to the example inputs. Further, in some examples, training machine learning algorithms using the training examples may generate a trained machine learning algorithm, and the trained machine learning algorithm may be used to estimate outputs for inputs not included in the training examples. In some examples, engineers, scientists, processes and machines that train machine learning algorithms may further use validation examples and/or test examples. For example, validation examples and/or test examples may include example inputs together with the desired outputs corresponding to the example inputs, a trained machine learning algorithm and/or an intermediately trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and/or test examples, the estimated outputs may be compared to the corresponding desired outputs, and the trained machine learning algorithm and/or the intermediately trained machine learning algorithm may be evaluated based on a result of the comparison. In some examples, a machine learning algorithm may have parameters and hyper parameters, where the hyper parameters are set manually by a person or automatically by an process external to the machine learning algorithm (such as a hyper parameter search algorithm), and the parameters of the machine learning algorithm are set by the machine learning algorithm according to the training examples. In some implementations, the hyper-parameters are set according to the training examples and the validation examples, and the parameters are set according to the training examples and the selected hyper-parameters., par. 80).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to include uses an algorithm implemented in one or more of: an artificial intelligence routine, a neural network routine, a machine learning routine, a deep learning routine, and any routine that simulates human methodology in order to use a neural network trained using example video frames including previously-identified surgical phases for image analysis as taught by Wolf (par. 160).
Regarding claim 9: Mitsuhashi satisfies all the elements of claim 1. Mitsuhashi further discloses wherein the display engine (When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44)., par. 105)
Mitsuhashi fails to specifically address uses algorithms comprised of one or more of: artificial intelligence routines, neural network routines, machine learning routines, deep learning routines, and any routines that simulate human methodology.
Wolf discloses uses an algorithm implemented in one or more of: an artificial intelligence routine, a neural network routine, a machine learning routine, a deep learning routine, and any routine that simulates human methodology (In some embodiments, machine learning algorithms (also referred to as machine learning models in the present disclosure) may be trained using training examples, for example in the cases described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regressions algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbors algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recursive neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and so forth. For example, a trained machine learning algorithm may comprise an inference model, such as a predictive model, a classification model, a regression model, a clustering model, a segmentation model, an artificial neural network (such as a deep neural network, a convolutional neural network, a recursive neural network, etc.), a random forest, a support vector machine, and so forth. In some examples, the training examples may include example inputs together with the desired outputs corresponding to the example inputs. Further, in some examples, training machine learning algorithms using the training examples may generate a trained machine learning algorithm, and the trained machine learning algorithm may be used to estimate outputs for inputs not included in the training examples. In some examples, engineers, scientists, processes and machines that train machine learning algorithms may further use validation examples and/or test examples. For example, validation examples and/or test examples may include example inputs together with the desired outputs corresponding to the example inputs, a trained machine learning algorithm and/or an intermediately trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and/or test examples, the estimated outputs may be compared to the corresponding desired outputs, and the trained machine learning algorithm and/or the intermediately trained machine learning algorithm may be evaluated based on a result of the comparison. In some examples, a machine learning algorithm may have parameters and hyper parameters, where the hyper parameters are set manually by a person or automatically by an process external to the machine learning algorithm (such as a hyper parameter search algorithm), and the parameters of the machine learning algorithm are set by the machine learning algorithm according to the training examples. In some implementations, the hyper-parameters are set according to the training examples and the validation examples, and the parameters are set according to the training examples and the selected hyper-parameters., par. 80).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to include uses an algorithm implemented in one or more of: an artificial intelligence routine, a neural network routine, a machine learning routine, a deep learning routine, and any routine that simulates human methodology in order to use a neural network trained using example video frames including previously-identified surgical phases for image analysis as taught by Wolf (par. 160).
Regarding claim 21: Mitsuhashi satisfies all the elements of claim 20. Mitsuhashi further discloses wherein the additional data generated by the image analysis engine includes image characteristics associated with a category (Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105) is related to one or more of: general image category (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105)
Mitsuhashi fails to specifically address associated with tissue.
Wolf discloses associated with tissue (As discussed above, an anatomical structure may be any particular part of a living organism, including, for example organs, tissues, ducts, arteries, cells, or other anatomical parts. The first set of frames may be analyzed to identify the anatomical structure using various techniques, for example as described above., par. 261).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to include associated with tissue in order to analyze frames to identify tissue as taught by Wolf (par. 261).
Regarding claim 23: Mitsuhashi satisfies all the elements of claim 20. Mitsuhashi further discloses wherein the additional data generated by the image analysis engine includes image characteristics associated with a category (Next, the display image generation process (step S3) illustrated in FIG. 6 will be described. FIG. 15 is a flowchart illustrating a procedure of the display image generation process illustrated in FIG. 6. The display image generation unit 434 acquires the first image to be displayed from images generated by the image generation unit 431 (step S41). The display image generation unit 434 acquires association information for the first image to be displayed, from the association unit 433 (step S42). The display image generation unit 434 determines whether there is a second image to be associated with the first image to be displayed, on the basis of the acquired association information (step S43). When it is determined that there is a second image to be associated with the first image to be displayed (step S43: Yes), the display image generation unit 434 acquires the second image associated with the first image from images generated by the image generation unit 431 (step S44). Composite image generation process is performed to generate a composite image in which the first image and the second image are combined (step S45), and the generated composite image is output as the display image to the control unit 44 (step S46)., par. 105)
Mitsuhashi fails to specifically address of anatomic sublocation.
Wolf discloses of anatomic sublocation (In some exemplary embodiments, the information that distinguishes portions of the historical surgical footage into frames associated with an intraoperative surgical event may include detected tools and anatomical features in associated frames. For example, the disclosed methods may include using an image and/or video analysis algorithm to detect tools and anatomical features. The tools may include surgical tools, as described above, or other nonsurgical tools. The anatomical features may include anatomical structures (as defined in greater detail above) or other parts of a living organism. The presence of both a surgical tool and an anatomical structure detected in one or more associated frames, may serve as an indicator of surgical activity, since surgical activity typically involves surgical tools interacting with anatomical structures. For example, in response to a detection of a first tool in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first tool in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In another example, in response to a detection of a first anatomical feature in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first anatomical feature in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In some examples, video footage may be further analyzed to detect interaction between the detected tools and anatomical features, and distinguishing the surgical activity from non-surgical activity may be based on the detected interaction. For example, in response to a detection of a first interaction in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first interaction in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In some examples, video footage may be further analyzed to detect actions performed by the detected tools, and distinguishing the surgical activity from non-surgical activity may be based on the detected actions. For example, in response to a detection of a first action in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first action in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event. In some examples, video footage may be further analyzed to detect changes in the condition of anatomical features, and distinguishing the surgical activity from non-surgical activity may be based on the detected changes. For example, in response to a detection of a first change in a group of frames, the group of frames may be determined to be associated with an intraoperative surgical event, while in response to no detection of the first change in the group of frames, the group of frames may be identified as not associated with the intraoperative surgical event., par. 203).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to include associated with tissue in order to anatomical features in frames as taught by Wolf (par. 203).
Regarding claim 27: Mitsuhashi satisfies all the elements of claim 26. Mitsuhashi further discloses wherein the procedure-related item (Fig. 15, YES or NO determination) is one of:
Mitsuhashi fails to specifically address is one of: a surgical tool, a procedural tool, a surgical instrument, a procedural instrument, a temporary implant, a permanent implant, a suture, a fastening device, and a cannula.
Wolf discloses is one of: a surgical tool, a procedural tool, a surgical instrument, a procedural instrument, a temporary implant, a permanent implant, a suture, a fastening device, and a cannula (At step 910, process 900 may include analyzing the particular surgical footage to detect a medical instrument. A medical instrument may refer to any tool or device used for treatment of a patient, including surgical tools, as described above. In addition to the surgical tools listed above, medical instruments may include, but are not limited to stethoscopes, gauze sponges, catheters, cannulas, defibrillators, needles, trays, lights, thermometers, pipettes or droppers, oxygen masks and tubes, or any other medical utensils., par. 206).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to include is one of: a surgical tool, a procedural tool, a surgical instrument, a procedural instrument, a temporary implant, a permanent implant, a suture, a fastening device, and a cannula in order to analyze a particular surgical footage to detect a medical instrument as taught by Wolf (par. 206).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLOTTE M BAKER whose telephone number is (571)272-7459. The examiner can normally be reached Mon - Fri 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER MEHMOOD can be reached at (571)272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLOTTE M BAKER/Primary Examiner, Art Unit 2664
12 June 2026
Prosecution Insights