Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This communication is filed in response to the action filed on 11/14/2024.
Claims 1-20 are currently pending.
Information Disclosure Statement
The information disclosure statements (IDS’s) filed on 11/14/2024; and 07/17/2026 have been fully considered.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-5, 7-20 are rejected under 35 § U.S.C. 102(a)(1) as being anticipated by US 2020/0273581 A1 to WOLF et al. (hereinafter “WOLF”).
As per claim 1, WOLF discloses a method for processing image data (a computing system adapted to perform operations related to a medical/surgical image processing method; abstract; fig 1; paragraphs [0083-0089]), the method comprising: providing source data from images acquired by a camera viewing an object (the computing system includes camera components115, 121, 123, and 125 attached to an endoscope and views anatomical objects of a subject anatomy; abstract; fig 1 and 29; paragraphs [0083-0089], [0584-0585]), providing target data related to the object (the computing system is adapted to provide recommended event data during a surgical procedure and is determined using the AI model; paragraphs [0487-0490]), operating an AI system comprising at least one artificial neural network (wherein said AI model of the computing system is based on a neural network framework; paragraphs [0080-0082], [0507], [0552]), wherein the source data are used as an input of the AI system (training data of historical surgical data stored in a database is used to train the model to identify recommended surgical event data; paragraphs [0080-0082], [0507], [0552]), and wherein the AI system is trained to identify a first correlation between the source data and the target data (wherein the computing system and resulting model are adapted to find the relationship between the historical data and the target region/real world data of the procedure being conducted; fig 6; paragraphs [0141], [0169-0175], [0188-0189], [0237-0239]), and processing the source data and the target data based on the first correlation to obtain a second correlation between the source data and the target data (and further the system processes the current data and the historical data to overlay historical data as a guide to the surgeon performing the current procedure in real time; paragraphs [0141], [0169-0175], [0188-0189], [0237-0239], [0568]).
As per claim 2, WOLF discloses the method of claim 1, wherein: the second correlation defines a spatial relation between the source data and the target data, the spatial relation describes at least one spatial transformation between at least a portion of the source data and at least a portion of the target data, and the transformation is one or more of a translation, a rotation, a reflection and a deformation (the computing system is adapted to perform the image translation on images captured by the cameras 115, 121, 123, and 125 and that translation is a rotation; paragraphs [0083-0087], [0307], [0321]), [0378]).
As per claim 3, WOLF discloses the method of claim 1, wherein: the source data relates to a portion of the object observed by the camera, the target data relates to a target area which is larger than the observed portion and includes the observed portion, and the AI system is trained to identify the first correlation as describing a position of the observed portion within the target area (the computing system comprising the AI model is adapted to take medical images of a full area (the entire image acting as the target area since the cameras are on an endoscope inside the subject anatomy) of the subjects anatomy and identify a smaller region of interest acting as the observed portion of the subjects anatomy receiving the surgical procedure; paragraphs [0087], [0400], [0463]).
As per claim 4, WOLF discloses the method of claim 3, wherein the processing of the source data and the target data to obtain the second correlation comprises: defining a crop region within the target area, the crop region extending around the position of the observed portion, and processing the target data of the crop region and the source data to obtain the second correlation (identifying the region of interest from the overall image of the subjects anatomy and zooming into the region of interest which acts substantially as cropping out the rest of the target area/initial image of the subjects anatomy; paragraphs [0083-0087], [0528-0534], [0748]).
As per claim 5, WOLF discloses the method of claim 1, wherein: the AI system is trained to recognize the presence of an object feature within at least one of the source data and the target data, and to determine positions of the object feature within both, the coordinate system of the source data and the coordinate system of the target data, and the object feature is a dye marking or a visible structural feature of the object (the computing system using the AI model is adapted to identify anatomical structures, markers, and surgical events related to the surgical procedure and further defines a coordinate system with the anatomy of the subject to provide location, position and orientation data; paragraphs [0109], [0165-0166], [0250], [0458], ).
As per claim 7, WOLF discloses the method of claim 1, wherein the target data are obtained from an imaging modality applied to the object (the subject data is obtained using imaging modalities such as MRI and CT scanning; paragraph [0183]).
As per claim 8, WOLF discloses the method of claim 7, wherein the imaging modality is one or more of Computer Tomography (CT) and Magnetic Resonance Imaging (MRI) (the subject data is obtained using imaging modalities such as MRI and CT scanning; paragraph [0183]; paragraph [0183]).
As per claim 9, WOLF discloses the method of claim 1, wherein the source data and the target data are processed for one of two-dimensional or three-dimensional display to be viewed by a user (the imaging data is processed and displayed over a display which provides 3D viewing; fig 1; paragraphs [0092], [0183], [0329, [0722]).
As per claim 10, WOLF discloses the method of claim 9, wherein the source data and the target data are processed for one of two-dimensional or three-dimensional display to be viewed by a user by using an augmented reality device (the imaging data is processed and displayed over a display which provides 3D viewing, and may also be implemented into an augmented reality device for augmented reality real time surgical operations; fig 1 and 29; paragraphs [0092], [0098], [0183], [0568], [0584]).
As per claim 11, WOLF discloses the method of claim 9, wherein the source data and the target data are displayed as being overlaid onto one another based on the second correlation (the historical data and the subject data are overlaid via the AI model of the computing system and displayed to the surgeon/user; fig 4; paragraphs [0109-0111], [0568]).
As per claim 12, WOLF discloses a system for processing image data (a computing system adapted to perform operations related to a medical/surgical image processing method; abstract; fig 1; paragraphs [0083-0089]), the system comprising: a camera configured to acquire images of an object, a source data processor configured to provide source data based on the images acquired by the camera (the computing system includes camera components115, 121, 123, and 125 attached to an endoscope and views anatomical objects of a subject anatomy and determines data based on a trained ai model trained on historical data; abstract; fig 1 and 29; paragraphs [0083-0089], [0584-0585]), a target data processor configured to provide target data related to the object (the computing system is adapted to provide recommended event data during a surgical procedure and is determined using the AI model; paragraphs [0487-0490]), an AI system configured to accept the source data as input (training data of historical surgical data stored in a database is used to train the model to identify recommended surgical event data; paragraphs [0080-0082], [0507], [0552]), wherein the AI system comprises at least one artificial neural network (wherein said AI model of the computing system is based on a neural network framework; paragraphs [0080-0082], [0507], [0552]) trained to identify a first correlation between the source data and the target data (wherein the computing system and resulting model are adapted to find the relationship between the historical data and the target region/real world data of the procedure being conducted; fig 6; paragraphs [0141], [0169-0175], [0188-0189], [0237-0239]), a registration processor configured to process the source data and the target data based on the first correlation to obtain a second correlation between the source data and the target data (and further the system processes the current data and the historical data to overlay historical data as a guide to the surgeon performing the current procedure in real time; paragraphs [0141], [0169-0175], [0188-0189], [0237-0239], [0568]).
As per claim 13, WOLF discloses the system according to claim 12, wherein the registration processor is configured to determine a transformation between at least a portion of the source data and at least a portion of the target data, wherein the transformation is at least one of or a combination of a translation, a rotation, a reflection and a deformation (the computing system is adapted to perform the image translation on images captured by the cameras 115, 121, 123, and 125 and that translation is a rotation; paragraphs [0083-0087], [0307], [0321]), [0378]).
As per claim 14, WOLF discloses the system according to claim 12, further comprising a cropping processor configured to define a crop region within the target area, the crop region extending around the position of the observed portion, and wherein the registration processor is configured to process the source data and the target data of the crop region to obtain the second correlation (identifying the region of interest from the overall image of the subjects anatomy and zooming into the region of interest which acts substantially as cropping out the rest of the target area/initial image of the subjects anatomy; paragraphs [0083-0087], [0528-0534], [0748]).
As per claim 15, WOLF discloses the system according to claim 12, further comprising a display configured for one of two-dimensional or three-dimensional display of the source data and the target data to be viewed by a user (the imaging data is processed and displayed over a display which provides 3D viewing; fig 1; paragraphs [0092], [0183], [0329, [0722]), wherein the display is configured to display the source data and the target data as being overlaid onto one another based on the second correlation (the historical data and the subject data are overlaid via the AI model of the computing system and displayed to the surgeon/user; fig 4; paragraphs [0109-0111], [0568]).
As per claim 16, WOLF discloses the system of claim 15, wherein the display comprises an augmented reality device configured for real-time display of the source data according to the images acquired by the camera (the imaging data is processed and displayed over a display which provides 3D viewing, and may also be implemented into an augmented reality device for augmented reality real time surgical operations; fig 1 and 29; paragraphs [0092], [0098], [0183], [0568], [0584]).
As per claim 17, WOLF discloses the system of claim 16, wherein the augmented reality device is further configured for real-time display of target information obtained from the target data as being overlaid onto the source data (the historical data and the subject data re overlaid via the AI model of the computing system and displayed to the surgeon/user; fig 4; paragraphs [0109-0111], [0568]).
As per claim 18, WOLF discloses an endoscope imaging system comprising (the system of fig 1 applied to an endoscopic surgical system; figs 1 and 29; paragraphs [0584-0585]): the system of claim 12 (see the full rejection of claim 12); and an endoscope, wherein the camera is an endoscope camera configured to be mounted to the endoscope (the computing system comprises an endoscope which comprises the cameras of fig 1 mounted to the endoscope; figs 1 and 29; paragraphs [0083-0087], [0584-0585]).
As per claim 19, WOLF discloses non-transitory computer-readable storage medium storing instructions for processing image data that cause a computer to at least perform (a computing system which comprises computing components such as a processor and memory to execute and store instructions, data, and programs respectively the programs and instructions related to the operations performed during a medical/surgical image processing method; abstract; fig 1; paragraphs [0083-0089]): providing source data from images acquired by a camera viewing an object (the computing system includes camera components115, 121, 123, and 125 attached to an endoscope and views anatomical objects of a subject anatomy; abstract; fig 1 and 29; paragraphs [0083-0089], [0584-0585]), providing target data related to the object (the computing system is adapted to provide recommended event data during a surgical procedure and is determined using the AI model; paragraphs [0487-0490]), operating an AI system comprising at least one artificial neural network (wherein said AI model of the computing system is based on a neural network framework; paragraphs [0080-0082], [0507], [0552]), wherein the source data are used as an input of the AI system (training data of historical surgical data stored in a database is used to train the model to identify recommended surgical event data; paragraphs [0080-0082], [0507], [0552]), and wherein the AI system is trained to identify a first correlation between the source data and the target data (wherein the computing system and resulting model are adapted to find the relationship between the historical data and the target region/real world data of the procedure being conducted; fig 6; paragraphs [0141], [0169-0175], [0188-0189], [0237-0239]), and processing the source data and the target data based on the first correlation to obtain a second correlation between the source data and the target data (and further the system processes the current data and the historical data to overlay historical data as a guide to the surgeon performing the current procedure in real time; paragraphs [0141], [0169-0175], [0188-0189], [0237-0239], [0568]).
As per claim 20, WOLF discloses a method for training an AI system comprising at least one artificial neural network (a computing system comprising a trainable AI model of the computing system and is based on a neural network framework and comprises a training method; paragraphs [0080-0082], [0507], [0552]), wherein the method comprises the steps of: providing a plurality of different training data sets (a machine learning model may be trained using training examples to determine recommendations based on information related to surgical decision making junctions trained using historical surgical data; paragraphs [0583], [0595-0598]), each training data set comprising a source data set obtained from at least one image acquired by a camera viewing an object (the training data sets comprising historical video data of prior surgical procedures from image data acquired using the cameras attached to the endoscope device; paragraphs [0583], [0595-0598]), a target data set related to the object, and a first correlation between the source data set and the target data set (wherein the computing system and resulting model are adapted to find the relationship between the historical data and the target region/real world data of the procedure being conducted; fig 6; paragraphs [0141], [0169-0175], [0188-0189], [0237-0239]), and training the AI system by inputting the training data sets into the AI system (training data of historical surgical data stored in a database is used as the input to train the model to identify recommended surgical event data; paragraphs [0080-0082], [0507], [0552]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claim 6 is rejected under 35 § U.S.C. 103 as being obvious over US 2020/0273581 A1 to WOLF et al. (hereinafter “WOLF”) in view of US 2023/0043645 A1 to PROTSENKO et al. (hereinafter “PROTSENKO”).
As per claim 6, WOLF discloses the method of claim 1, wherein: the source data includes spatial coordinates of a plurality of source points defining a source point cloud, the target data includes spatial coordinates of a plurality of target points defining a target point cloud (the computing system determines the source data by using a point cloud and spatial coordinate system in order to identify coordinates of points in the cloud; paragraphs [0083-0085], [0092]), the point cloud registration using a feature matching algorithm (during point cloud registration the AI model of the computing system is used which comprises matching algorithms; paragraphs [0083-0085], [0092], [0602]). WOLF fails to disclose and the processing of the source data and the target data to obtain the second correlation comprises a point cloud registration which estimates a transformation matrix between at least a portion of the source point cloud and at least a portion of the target point cloud.
PROTSENKO discloses and the processing of the source data and the target data to obtain the second correlation comprises a point cloud registration which estimates a transformation matrix between at least a portion of the source point cloud and at least a portion of the target point cloud (the computing system is adapted to perform translations by translating the points by using a data structure, applied in this example as a matrix, with predefined constants for every transition/translation; abstract; paragraph [0140]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify WOLF to have a point cloud registration which estimates a transformation matrix of PROTSENKO reference. The Suggestion/motivation for doing so would have been to provide an evaluation to determine if a detected bounding box in a certain image depicts the same polyp as in another box in one or more preceding images, as suggested by paragraph [0141] of PROTSENKO. Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine PROTSENKO with WOLF to obtain the invention as specified in claim 6.
Conclusion
Examiner's Note: Examiner has cited figures, and paragraphs in the references as applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested for the applicant, in preparing the responses, to fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. Examiner has also cited references in PTO892 but not relied on, which are relevant and pertinent to the applicant’s disclosure, and may also be reading (anticipatory/obvious) on the claims and claimed limitations. Applicant is advised to consider the references in preparing the response/amendments in-order to expedite the prosecution.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. These prior arts include the following:
US 7,684,647 B2
US 2024/0161928 A1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVIN JACOB DHOOGE whose telephone number is (571) 270-0999. The examiner can normally be reached 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800- 786-9199 (IN USA OR CANADA) or 571-272-1000.
/D J DHOOGE/Examiner, Art Unit 2677
Prosecution Insights