DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 7, filed 12/29/2025, with respect to the specification objection have been fully considered and are persuasive. The objection of the specification has been withdrawn.
Applicant’s arguments with respect to claim(s) 1-5 have been considered but are moot because the new ground of rejection does not rely on all references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The argument states that the previously applied references do not disclose the features of “classify an gastrointestinal anatomical position of the video frame by identifying a part corresponding to a preset gastrointestinal anatomical position class inside a stomach in the video frame, wherein the preset anatomical position class includes at least one of an anterior wall (AW), a posterior wall (PW), a greater curvature (GC), or a lesser curvature (LC) for each of a body of the stomach, an angle of the stomach, or an antrum of the stomach.” The deficiency of the previously applied references is cured by the Kim reference and will be briefly explained why below.
Regarding the Kim reference, this reference discloses using machine learning models to classify the position of a frame captured of the stomach. The specific classification can relate to the antrum or the angle of the stomach, which is taught in ¶ [23] and [82]. The invention further discloses classifying areas, such as the anterior or posterior wall or greater curvature in order to segment a specific area to display a location of a particular lesion, which is taught in ¶ [86] and [87]. Therefore, these aspects of the reference of Kim, combined with the aspects of Kim ‘771 to display a stomach shaped display of an area, disclose the features of the independent claim when combined with the previously applied reference.
Thus, based on the above, the features of the claims are disclosed below.
Election/Restrictions
Newly submitted claims 12-15 are directed to an invention that is independent or distinct from the invention originally claimed for the following reasons: Claims 12-15 are directed towards the features in the non-elected invention in claims 6-11 that were cancelled.
Since applicant has received an action on the merits for the originally presented invention, this invention has been constructively elected by original presentation for prosecution on the merits. Accordingly, claims 12-15 are withdrawn from consideration as being directed to a non-elected invention. See 37 CFR 1.142(b) and MPEP § 821.03.
To preserve a right to petition, the reply to this action must distinctly and specifically point out supposed errors in the restriction requirement. Otherwise, the election shall be treated as a final election without traverse. Traversal must be timely. Failure to timely traverse the requirement will result in the loss of right to petition under 37 CFR 1.144. If claims are subsequently added, applicant must indicate which of the subsequently added claims are readable upon the elected invention.
Should applicant traverse on the ground that the inventions are not patentably distinct, applicant should submit evidence or identify such evidence now of record showing the inventions to be obvious variants or clearly admit on the record that this is the case. In either instance, if the examiner finds one of the inventions unpatentable over the prior art, the evidence or admission may be used in a rejection under 35 U.S.C. 103 or pre-AIA 35 U.S.C. 103(a) of the other invention.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: reception interface in claim 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang (US Pub 2023/0154580 (Prov. Appl date: 11/18/2021)) in view of Park (US Pub 2011/0301447), Kim (US Pub 2023/0301503 (PCT filed: 8/4/2021)) and Kim ‘771 (KR 102268771 (Pub Date: 6/24/2021)).
Re claim 1: (Currently Amended) Wang discloses a gastroscopic image diagnosis supporting system for supporting diagnosis of a medical image, the gastroscopic image diagnosis supporting system comprising a computing system,
wherein the computing system comprises:
a reception interface (interpretation: In this case, the artificial intelligence workstation 220 may include an input/reception interface module (not shown) configured to receive a gastroscopic image (or a captured image) received from the gastroscopic image acquisition module 232, which is taught in ¶ [84]. This interpretation and its equivalents are utilized for this claim term hereinafter in the Office Action.) configured to receive a gastroscopic image as the medical image (e.g. images from the endoscope of the GI tract are received, which is taught in ¶ [37].);
[0037] FIG. 1A shows an example of a data processing system 100 configured to execute one or more processes for automatically generating structured data records, such as electronic medical records, from endoscopy data associated with an endoscopic procedure. The data processing system 100 is configured to receive endoscopy data that includes images and/or video data. The data processing system 100 processes the endoscopy data and extracts features from the endoscopy data. Based on the extracted features, the data processing system 100 is configured to determine a precise location in the GI tract of the patient that is represented in images or video data of the endoscopy data. The precise location is labeled, and the label is stored in a node of the data store with the video or image data that include the precise location. In some implementations, the label is a time stamp that indicates where in a video of the endoscopy a particular feature is represented. In some implementations, the label specifies the precise location and/or GI tract feature represented in an image. The label is a portion of anatomical location data stored in the node with the video and image data for the endoscopy of the patient. In some implementations, each node in the data store is keyed to a patient identifier, and each node stores data entries representing a plurality of endoscopies for the patient.
memory or a database configured to store one or more medical image analysis models each having a function of analyzing the gastroscopic image (e.g. a memory is used to store one or more machine learning models that analyze the images input, which is taught in ¶ [43] and [53].); and
[0043] The data processing system includes a processing device 110, a memory 111 (or a computer readable hardware storage device) configured to host instructions 112, a machine learning module configured to execute one or more trained machine learning platforms, a feature extraction or image processing module 114 to process endoscopy images and/or video, and a location detection module 115. Generally, endoscopy data includes data generated from or associated with an endoscopic procedure and can include video data 102 and associated metadata 104. The data processing system 100 can include an endoscopic processing unit 120 configured to connect to an endoscope 108 for capturing image and/or video data generated by an imaging device of the endoscope 108 during the endoscopy. Instructions can be hosted in either volatile or non-volatile memory, or a combination thereof.
[0053] The feature extraction module 114 is configured to process the image or video data 102 and other data associated with the endoscopy of the endoscopy data 102 to extract data for populating fields of the target EMR 106. In some implementations, the image processing module 114 is a part of a machine learning module, wherein the image processing module extracts data from the images or videos, and the machine learning module performs classification of the extracted data. For example, the image processing module 114 may perform thresholding operations or feature extraction based on signals received from the machine learning module (e.g., setting threshold values or identifying features in the images to extract).
[0061] The processing device 110 can be communicatively coupled to the endoscopic processing unit 120 and configured to receive spatially arranged image data (e.g., video data) corresponding with one or more images captured by the imaging device. In some implementations, the processing device 110 includes a general purpose processor. In some implementations, the processing device 110 includes at least one applicable inference processor, accelerated processor which can be utilized in half, single, or double precision (16, 32, or 64 bit floating-point) calculation. The computer processor 110 can also include lots of compute unified device architecture (CUDA) cores, etc., or a combination of thereof. In some implementations, the computer processors 110 include a central processing unit (CPU). In some implementations, the processing device 110 includes at least one application specific integrated circuit (ASIC). The processing device 110 can also include general purpose programmable microprocessors, special-purpose programmable microprocessors, digital signal processors (DSPs), programmable logic arrays (PLAs), field programmable gate arrays (FPGA), special purpose electronic circuits, etc., or a combination thereof. The processing device 110 is configured to execute program code means such as the computer-executable instructions 112.
a processor (e.g. the system contains a processor, which is taught in ¶ [61] above.), wherein the processor is configured to:
analyze a video frame of the gastroscopic image using a first medical image analysis model of the medical image analysis models (e.g. the machine learning models are used to analyze the data in order to perform classification of the received data, which is taught in ¶ [43] and [53] above. For example, a first machine learning model can be used to analyze the input data, which is taught in ¶ [39].);
[0039] Generally, the data processing system 100 is configured to generate the anatomical location data and export it to the data store in a series of stages. A first stage includes applying, by the data processing system 100, a single machine learning model or combination of machine learning models to analyze data from the endoscopic processing unit and metadata from a patient. The analysis is described in further detail with respect to FIG. 1B.
classify an anatomical position of the video frame by identifying a part corresponding to a preset anatomical position class in the video frame (e.g. the landmark detection module is a machine learning model used to detect landmarks to classify a precise location of the image or frame in the video, which is taught in ¶ [72]-[74].); and
[0072] FIG. 1C shows sub-modules for the location detection module 115 including machine learning sub-modules. The machine learning sub-modules include a landmark detection module 160, a boundary detection module 162, and a motion detection module 164. These modules 160, 162, and 164 can work independently or in conjunction with each other to estimate the anatomical location of an image or frame in a video. In some implementations, a score is tied to the prediction of the anatomical locations determined by each module 160, 162, 164. This score may be a confidence score that is determined by the machine learning algorithms present in each module. For example, a weighted vote can be performed based on these confidence scores.
[0073] The landmark detection module 160 includes logic executed by the data processing system 100 for performing landmark detection within the colon. The landmark detection module 160 is configured to detect unique landmarks within the anatomical locations or regions to determine the broader region or a granular, precise location. The landmark detection module 160 is configured to detect features including one of an appendix, a cecum, an ileocecal valve, an ileum, a Taenia coli, a right colic (hepatic) flexure, a left colic (splenic) flexure, a haustra, a sigmoid flexure, a rectum, and an anus. For example, the cecum contains three possible landmarks, including an ileocecal valve, an appendiceal orifice, and Y-shaped “crow's feet” folds. Following this example, the detection of a single landmark, or combination of landmarks within an image or frame within a video, the landmark detection module 160 is able to determine that the anatomical location of this specific image or frame is the cecum.
[0074] In some cases, there are no unique landmarks present in a segment of the colon being analyzed in the endoscopy data 102. For example, the transverse and ascending colon are relatively similar and do not have unique anatomical landmarks that differentiate both segments. To assist the landmark detection module 160 to accurately predict the anatomical location without unique landmarks, the boundary detection module 162 is executed.
store information about the classified anatomical position of the video frame and the identified anatomical position class in the video frame as index information of the finding information together with the finding information when a user captures and stores based on a capturing and storing the video frame as the finding information about a lesion in the gastroscopic image by a user (e.g. the system stores label text within metadata with videos or images regarding an anatomical location of an image feature. In addition, the information about the detection of polyps can be stored with the metadata information. The metadata can be searched in order to retrieve the polyp information and the video data associated therewith. The searchable information can be considered as the finding information, and the polyps are considered as lesions. The above is explained in ¶ [89]-[98].);
[0089] Returning to FIG. 1A, the data processing system 100 is configured to label the images or videos and/or export the relevant labels into a data store 106 for retrieval at a later stage. In some implementations, the labels will be displayed as timestamps for videos. For example, at 1 min 30 s into a given video, the data processing system 100 is able to predict and determine that a splenic flexure was detected. The data processing system 100 can compile the timestamps into a list.
[0090] In some implementations, the data processing system 100 includes a user interface that displays the anatomical location of the image/frame in the video in text format. As a user observes and advances the video, an anatomical location displayed in text adjusts accordingly. In some implementations, the user interface includes a confidence level output by the machine learning model for determining the anatomical location. In some implementations, alternative anatomical location recommendations having lower confidence or accuracy scores are displayed in a separate region of the user interface (e.g., a text box).
[0091] In some implementations, the data processing system 100 generates bounding boxes to highlight key anatomical landmarks that resulted in the prediction of the anatomical location. For example, if the data processing system 100 detects a crow's feet anatomical landmark, a bounding box and a text label are generated and dynamically displayed over the landmark in the video. In some implementations, the labels are metadata that are searchable and responsive to data queries of the EMR data 106.
[0092] The data processing system 100 is configured to export relevant labels of images and videos into a data warehouse, such as EMR system 130. These data can be retrieved at a later stage or combined with alternative data sources (e.g., information about detection of polyps). In some implementations, the generated anatomical locations data are transcribed into a patient's EMR/EHR directly (e.g., using interface 132/134). In some implementations, the data are stored elsewhere with some association with the image/series of images or video, such as in a node.
[0093] FIG. 2 shows an example process for generating anatomical location data from endoscopy data. Generally, the data processing system (e.g., the data processing system 100 of FIG. 1) is configured to intake data for colonoscopy procedure or similar endoscopic procedure. The data processing system 100 is configured to generate anatomical location labels based on images 102 and metadata 104. As shown in FIG. 2, the images and metadata 202 of the endoscopy are transformed into anatomical location data 208.
[0094] The images and metadata 202 are received from the endoscopic processing unit 120. The feature extraction module 114 extracts feature data 204 from the video and image data 102 and the metadata 104. Here, the features are represented as blocks “A,” “B,” and “C.” The feature data 204 are generated based on events of the endoscopy, and these features can be combined to generate further data. For example, a start event corresponds to a start time feature, and a stop event corresponds to a stop time feature of the feature data 204. The duration of the procedure can be calculated by combining these two individual features.
[0095] As features are detected in the video data (e.g., a polyp is detected), the associated metadata (e.g., time elapsed for that image frame) can be associated with a newly extracted feature representing the polyp. For example, as a series of polyps are detected in the video data 102, each polyp is extracted as a feature (also called an event) of the feature data 204. In FIG. 2, features A, B, and C can each represent a polyp detected in the image data 102. Each feature A, B, and C is associated with other data that are determined from analysis of the video data. For example, for the third polyp (e.g., feature or event “C”), a size value is determined by processing of the images including the third polyp detected. For example, for each video frame including the third polyp “C,” a size of the polyp is estimated (e.g., by border detection that is converted to a size based on a known distance from the polyp to the camera or a similar calculation). A series of frames shows the polyp with an average size when viewed from various angles. The processing device 110 stores the size average as feature data which can be used to populate the EMR 208 if requested by the EMR system 130.
[0096] Continuing with this example, transformed feature data 206 are generated from the processed image and video data 102 and metadata 104. For feature C, representing a third polyp, a time of detection is stored “Polyp 3 detected at 15 minutes into endoscopy.” For feature C, a location is stored “Polyp 3 is at lower GI tract.” For feature C, the size of the polyp is stored “Polyp 3 is 2 millimeters (mm) by 11 mm.”
[0097] Anatomical location data includes fields 1-3 each including respective values 208a-d that are generated from the respective transformed features 206a-c. Generally, the data included in the EMR 208 are determined based on a query configured by the EMR system 130, as subsequently described. In some implementations, the data processing system 100 is configured to determine what fields are included in EMR 208 and generate corresponding data to satisfy those fields with generated values.
[0098] The values 208a-c of the respective fields 1-3 of the anatomical location data 208 can be based on the transformed features but need not be identical to the transformed features. In some implementations, features A, B, and C can be directly imported into the anatomical location data 208. The report generated by the data processing system 100 can be formatted based on the query being generated by the EMR system 130. For example, when a size of the polyp is requested, the data processing system 100 can provide a numerical type output, a plain text stream output, and so forth based on the query.
display a user interface for accessing finding information of previous gastroscopy (e.g. the system displays anatomical location in the video in text format, which is taught in ¶ [89]-[91] above.).
However, Wang ‘580 fails to specifically teach the features of display a user interface for accessing finding information of previous gastroscopy on a first anatomical position map at an anatomical position of the finding information of the previous gastroscopy; and
provide the first anatomical position map to the user.
However, this is well known in the art as evidenced by Park. Similar to the primary reference, Park discloses a visual system of a GI tract (same field of endeavor or reasonably pertinent to the problem).
Park discloses display a user interface for accessing finding information of previous gastroscopy on a first anatomical position map at an anatomical position of the finding information of the previous gastroscopy (e.g. the invention discloses a digital colon that contains an anatomical position map of a colon that contains a piece of information that is associated with a previous colon video for review associated at a location on the digital colon, which is taught in ¶ [190] and [191]. This digital colon, in combination with the previously applied reference, would result in a record of a polyp and past scan being associated with a location within a part of the GI tract.); and
Digital Colon Model
[0190] A digital colon model is a visualization tool that enables standardized navigation through colon videos, as illustrated in FIG. 7. Starting with a generic colon model as illustrated in FIG. 7(a) (preferably, as illustrated in FIG. 6, consisting of the five anatomical colon segments of the rectum (10), sigmoid colon (11), descending colon (12), transverse colon (13), ascending colon (14), and cecum (15), and anchored by the anatomical colon landmarks of the anus (20) sigmoid/descending colon transition (21), splenic flexure (22), hepatic flexure (23), and ileocecal valve (24)) the video data as illustrated in FIG. 7(b) are mapped and superimposed onto the geometry of this generic model. While viewing the video data, either in real-time during a clinical exam or as part of a video review, an icon in the colon model (see FIG. 7(a)) depicts the estimated location of the colonoscope tip (100) within the colon.
[0191] This digital colon model is a standardized visualization tool for colonoscopy because every exam video can be superimposed onto the generic colon model. Furthermore, the digital colon model can help the physician to plan their treatment during the examination of the colon. For example, during entry, the physician can mark suspicious locations on the digital colon model. During withdrawal, the physician can be alerted to previously digitally marked regions and perform treatment. Additionally, for high-risk patients that require surveillance, the model can provide a framework for registering the patient's clinical state across exams, thereby enabling change detection.
[0192] The concept of the digital colon model can be augmented by, and in addition to, video data acquired using different macroscopic imaging modalities, including data from microscopic and spectroscopic probe systems, such as confocal microscopy, optical coherence tomography, and infrared spectroscopy. These technologies provide imaging or spectral information about the tissue on a microscopic scale.
provide the first anatomical position map to the user (e.g. the map of the anatomical position of the colon is provided to a user via a display, which is taught in ¶ [190] and [191] above.).
Therefore, in view of Park, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of display a user interface for accessing finding information of previous gastroscopy on a first anatomical position map at an anatomical position of the finding information of the previous gastroscopy; and provide the first anatomical position map to the user, incorporated in the device of Wang, in order to display information associated with a previous examination on an GI anatomical map, which aid a physician in their treatment of a detected anomaly (as stated in Park ¶ [191]).
However, the combination above fails to specifically teach the features of classify an anatomical position of the video frame by identifying a part corresponding to a preset anatomical position class inside a stomach in the video frame, wherein the preset anatomical position class includes at least one of an anterior wall (AW), a posterior wall (PW), a greater curvature (GC), or a lesser curvature (LC) for each of a body of the stomach, an angle of the stomach, or an antrum of the stomach, wherein the first anatomical position map is stomach-shaped.
However, this is well known in the art as evidenced by Kim. Similar to the primary reference, Kim discloses classification of stomach issues (same field of endeavor or reasonably pertinent to the problem).
Kim discloses classify an anatomical position of the video frame by identifying a part corresponding to a preset anatomical position class inside a stomach in the video frame, wherein the preset anatomical position class includes at least one of an anterior wall (AW), a posterior wall (PW), a greater curvature (GC), or a lesser curvature (LC) for each of a body of the stomach, an angle of the stomach, or an antrum of the stomach (e.g. the system discloses classifying a position within an image by identifying a part of the anatomical position class inside of a stomach, which is taught in ¶ [23] and [82]. The class can be the artium or angle of the stomach. Moreover, the system can classify and segment specific areas in the stomach for a lesion, such as the anterior, posterior walls, or greater curvature, which is taught in ¶ [86] and [87].),
[0023] In addition, when the anatomical location of the stomach is automatically classified and recognized with respect to the image captured during photographing using the plurality of image classification models and the location of the lesion is automatically stored in step d), the oral cavity/the laryngopharynx, the esophagus, and the esophagogastric junction may be classified and recognized using the first, second, and third classification models, the esophagogastric junction, the esophagus, and the body of the stomach may be classified and recognized using the second, third, and fourth classification models, the body of the stomach, the esophagogastric junction, and the antrum of the stomach may be classified and recognized using the third, fourth, and fifth classification models, the antrum of the stomach, the body of the stomach, the angle of the stomach, and the duodenal bulb may be classified and recognized using the fourth, fifth, eighth, and ninth classification models, and the duodenal bulb, the antrum of the stomach, and the second part of the duodenum may be classified and recognized using the fifth, ninth, and tenth classification models in order to automatically store the location of the lesion during the photographing process from the oral cavity and the laryngopharynx to the second part of the duodenum.
[0082] During execution of step S104, an image of a specific part in the captured gastroscopic image is segmented for each region by a region segmentation model and the same number of segmented maps as the number of target classes is output as the result of segmentation (step S105). That is, in the present embodiment, the region segmentation model is trained to segment an input image into a maximum of four regions. As shown in FIG. 8, therefore, the region segmentation model receives a color gastroscopic image as an input image and outputs the same number (4) of segmented maps as the number (4) of the target classes as resultant images. Here, the image of the specific part may include images of the body of the stomach (A), the antrum of the stomach (B), and the fundus of the stomach (C), as shown in FIGS. 9A, 9B, and 9C. At this time, each of the images of the body of the stomach (A), the antrum of the stomach (B), and the fundus of the stomach (C) may be segmented for each region by the region segmentation model, and may include an anterior wall, a posterior wall, a lesser curvature, and a greater curvature as the result of segmentation. For example, when the image of the specific part is the body of the stomach, the region segmentation model may be operated, and the anterior wall, the posterior wall, the lesser curvature, and the greater curvature may be displayed on a screen. When the lesion is recorded, the body of the stomach—the anterior wall or the body of the stomach—the anterior wall and accessory muscle connected to the greater curvature may be displayed.
[0086] Referring to FIG. 6, which shows that, in step S104 of FIG. 1, the anatomical location of the stomach is automatically classified and recognized using the plurality of image classification models and the location of the lesion is automatically reported, when a captured image is input, it is recognized whether the part in the input image is the second part of the duodenum, the duodenal bulb, or the antrum of the stomach using the fifth, ninth, and tenth models (steps S601 to S604). Upon determining in step S604 that the part is not the antrum of the stomach, the procedure returns to step S601, and upon determining that the part is the antrum of the stomach, it is recognized whether the part is the antrum of the stomach, the duodenal bulb, the angle of the stomach, or the body of the stomach using the fourth, fifth, eighth, and ninth models (steps S605 to S609). Upon determining in step S607 that the part is the duodenal bulb, the procedure returns to step S601, and upon determining in step S608 that the part is the angle of the stomach, the procedure returns to step S605. In addition, upon determining in step S609 that the part is not the body of the stomach, the procedure returns to step S605, and upon determining that the part is the body of the stomach, it is recognized whether the part is the body of the stomach, the antrum of the stomach, the fundus of the stomach, the cardia of the stomach, or the esophagogastric junction using the third, fourth, fifth, sixth, and seventh models (steps S610 to S615). Upon determining in step S612 that the part is the antrum of the stomach, the procedure returns to step S605, upon determining in step S613 that the part is the fundus of the stomach, the procedure returns to step S610, and upon determining in step S614 that the part is the cardia of the stomach, the procedure returns to step S610. In addition, upon determining in step S615 that the part is not the esophagogastric junction, the procedure returns to step S610, and upon determining that the part is the esophagogastric junction, it is recognized whether the part is the esophagogastric junction, the body of the stomach, or the esophagus using the second, third, and fourth models (steps S616 to S619). Upon determining in step S618 that the part is the body of the stomach, the procedure returns to step S610. In addition, upon determining in step S619 that the part is not the esophagus, the procedure returns to step S616, and upon determining that the part is the esophagus, it is recognized whether the part is the esophagus, the esophagogastric junction, or the cavity/the laryngopharynx using the first, second, and third models (steps S620 to S623). Upon determining in step S622 that the part is the esophagogastric junction, the procedure returns to step S616, upon determining in step S623 that the part is not the cavity/the laryngopharynx, the procedure returns to step S620, and upon determining that the part is the cavity/the laryngopharynx, it is determined whether inspection has been finished (step S624). Upon determining that inspection has not been finished, the procedure returns to step S620, and upon determining that inspection has been finished, the operation sequence is finished.
[0087] In the above series of processes, each of the body of the stomach, the antrum of the stomach, and the fundus of the stomach may be minutely segmented into the anterior wall/the posterior wall and the lesser curvature/the greater curvature in order to display the location of the lesion, through region segmentation, not image classification.
Therefore, in view of Kim, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of classify an anatomical position of the video frame by identifying a part corresponding to a preset anatomical position class inside a stomach in the video frame, wherein the preset anatomical position class includes at least one of an anterior wall (AW), a posterior wall (PW), a greater curvature (GC), or a lesser curvature (LC) for each of a body of the stomach, an angle of the stomach, or an antrum of the stomach, incorporated in the device of Wang, as modified by Park, in order to automatically recognize an anatomical location of a stomach in a gastroscopy, which can inform a medical professional of a situation to improve the safety of inspection (as stated in Kim ¶ [29] and [30]).
However, the combination above fails to specifically teach the feature of wherein the first anatomical position map is stomach-shaped.
However, this is well known in the art as evidenced by Kim ‘771. Similar to the primary reference, Kim ‘771 discloses displaying parts of the stomach (same field of endeavor or reasonably pertinent to the problem).
Kim ‘771 discloses wherein the first anatomical position map is stomach-shaped (e.g. in figure 6, a stomach-shaped image is shown on a display to show an anatomical position, which is taught in ¶ [69]-[73].).
[0070] In FIG. 6 , the control unit 121 generates two identical indicators corresponding to image processing information together with the image of the front or rear wall of the digestive tract and the image of the cut-out of the digestive tract, and the display unit 124 ) may display either one of the two indicators on the image of the anterior wall or the image of the posterior wall, and display the other one of the two indicators on the image of the shape in which the digestive tract is cut. According to FIG. 6, when it is difficult to accurately determine whether the lesion of the corresponding indicator occurred in either the anterior wall or the posterior wall only with the image of the incised shape of the digestive tract, the image of the anterior wall or the posterior wall can also be referred to, so that the accurate location can be identified.
[0071] Meanwhile, even in the case of FIG. 6 , at least one of the blood vessel and lymph node information described in FIG. 4 may be further displayed and provided to the user.
[0072] Also, in FIG. 6 , the color, size, pattern, etc. of each indicator may be implemented differently.
[0073] In the case of FIG. 6 , the position of each lesion of the digestive system is confirmed in two types of different image forms, so that it can be confirmed more precisely and in more detail, as well as more intuitively.
Therefore, in view of Kim ‘771, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the first anatomical position map is stomach-shaped, incorporated in the device of Wang, as modified by Park and Kim, in order to show a stomach-shaped image to a user, which allows for identification of a lesion in a specific area of the stomach more precisely (as stated in Kim ‘771 ¶ [73]).
Re claim 2: (Original) Wang discloses the gastroscopic image diagnosis supporting system of claim 1, wherein the processor is further configured to:
when the user requests checking of the finding information after gastroscopy (e.g. the user can search the labels within the metadata information to acquire the image or video associated with the anatomical location, which is taught in ¶ [89]-[92] above.).
However, Wang fails to specifically teach the wherein the processor is further configured to: provide a second anatomical position map that is stomach-shaped;
display the anatomical position of the video frame on the anatomical position map and provide the anatomical position map together with the video frame to the user while the video frame is displayed to the user through a user display; and
display an anatomical position of the finding information on the second anatomical position map and provide the second anatomical position map to the user when the user requests checking of the finding information after gastroscopy.
However, this is well known in the art as evidenced by Park. Similar to the primary reference, Park discloses a visual system of a GI tract (same field of endeavor or reasonably pertinent to the problem).
Park discloses provide a anatomical position map (e.g. the map of the anatomical position of the colon is provided to a user via a display, which is taught in ¶ [190] and [191] above.);
display the anatomical position of the video frame on the anatomical position map and provide the anatomical position map together with the video frame to the user while the video frame is displayed to the user through a user display (e.g. the user is shown a digital colon, which is considered as the GI anatomical position map, along with the video of the section where a suspicious portion within the colon can be marked for being suspicious. The video along with the digital colon are shown at the same time to the user when going over a video review of the colon, which is taught in ¶ [190] and [191] above.); and
display an anatomical position of the finding information on the anatomical position map and provide the anatomical position map to the user when the user requests checking of the finding information after gastroscopy (e.g. when interacting with the system, the user can see where in the colon area where a specific suspicious area icon is located once the area is interacted with, which is taught in ¶ [190] and [191] above. Combining these features with the aspect of Wang performs the features of the claim.).
Therefore, in view of Park, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of provide a anatomical position map; display the anatomical position of the video frame on the anatomical position map and provide the anatomical position map together with the video frame to the user while the video frame is displayed to the user through a user display; and display an anatomical position of the finding information on the anatomical position map and provide the anatomical position map to the user when the user requests checking of the finding information after gastroscopy, incorporated in the device of Wang, in order to display information associated with a previous examination on an GI anatomical map, which aid a physician in their treatment of a detected anomaly (as stated in Park ¶ [191]).
However, the combination above fails to specifically teach the feature of second anatomical position map.
However, this is well known in the art as evidenced by Kim ‘771. Similar to the primary reference, Kim ‘771 discloses displaying parts of the stomach (same field of endeavor or reasonably pertinent to the problem).
Kim ‘771 discloses second anatomical position map (e.g. the display in figure 6 shows a second anatomical position map that is also stomach-shaped. This is explained in ¶ [70]-[73] above.).
Therefore, in view of Kim ‘771, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of second anatomical position map, incorporated in the device of Wang, as modified by Park and Kim, in order to show a stomach-shaped image to a user, which allows for identification of a lesion in a specific area of the stomach more precisely (as stated in Kim ‘771 ¶ [73]).
Re claim 3: (Original) Wang discloses the gastroscopic image diagnosis supporting system of claim 1, wherein the processor is further configured to:
analyze the video frame of the gastroscopic image by using a second medical image analysis model of the medical image analysis models (e.g. the feature extraction module can be used to analyze the image through extracting features. The feature extraction module is comprised of machine learning modules, which is taught in ¶ [68].);
[0056] The feature extraction module 114 can process groups of frames or images of video data 102 together. The feature extraction module 114 can be configured to analyze the image in the context of a previous frame (or series of frames) or a subsequent frame (or series of frames). The feature extraction module 114 is configured to facilitate extraction and/or recognition, from image data, of features that are used for populating fields of the target EMR 106. For example, the feature extraction module 114 can facilitate detection of bleeding, polyp formation, etc. by applying one or more feature extraction processes using image processing, and populate an EMR field requesting a list of symptoms, a number of polyps detected, and so forth. Image processing processes can include object detection, pixel thresholding, application of filters to the images or portions of the images, and so forth.
[0068] FIG. 1B shows sub-modules of the feature extraction module for performing operations of the first stage including feature extraction. As previously described, the data processing system 100 is configured to use a single or combination of machine learning models to analyze data from the endoscopic processing unit 120 and metadata 104 from a patient. The data from the endoscopic processing unit 120 may come in the form of images or videos to form endoscopy data 102. In some implementations, only image and/or video data are processed. In some implementations, both image and/or video data and metadata 104 are processed for feature extraction.
detect whether a region suspected of being a lesion is present in the video frame (e.g. regions can be detected as having or not having an anomaly, such as a polyp. This is taught in ¶ [137] and [138].); and
[0137] The computations (e.g., dot product computations) for a convolutional layer, or other layers, of a neural network involve performing mathematical operations, e.g., multiplication and addition, using a computation unit of a hardware circuit of the model. The design of a hardware circuit can cause a system to be limited in its ability to fully utilize computing cells of the circuit when performing computations for layers of a neural network.
[0138] Based on the aforementioned techniques, the model is configured to identify locations of potential malignancies in images. In some implementations, potential malignancies include polyps. In some implementations, given a set of images, the model is capable of correctly detecting at least 87% of all polyps shown (e.g., at least one image of at least 87% of the polyps presented in the set of images will be correctly detected and identified). In some implementations, when given a set of images, and the model is capable of making a determination that an image does not contain a polyp, and that determination is correct at least 98.7% of the time (e.g., it is likely to be correct 98.7% of the times the machine learning 113 system makes a “does not contain polyp” classification).
classify a type of lesion by comparing the region suspected of being a lesion against lesion classes by using a third medical image analysis model of the medical image analysis models (e.g. the machine learning system (113) performs the classification of a lesion, or a polyp, when using a given set of images, which is taught in ¶ [138] above, [55] and [144].).
[0055] The feature extraction module 114 generally is configured to tag or identify images or frames as representing a symptom. However, how the image processing module 114 identifies the symptoms can be changed or updated based on feedback from the machine learning module 113. For example, the image processing module 114 can extract image data based on thresholds set or adjusted by the machine learning module. In some implementations, the machine learning module is configured to update, based on training data, image signature data used for classification of the image or video data.
[0064] As previously described, types of information can include start/stop times for a procedure, how many polyps were found and one or more characteristics of them. In some implementations, a human in the loop is still included to validate the generated record. Types of data can include when polyps appear and when they disappear in the video. The data processing system 100 is configured to capture images of polyps and potentially upload them to the EMR 106. In some implementations, the data processing system 100 is configured to detect an instrument used during the procedure, such as forceps or a snare. For example, the data processing system 100 is configured to remove a bounding box generated around a polyp when a surgeon is acting on it. The data processing system 100 is configured to detect when in a surgery phase and detect a withdrawal phase of the procedure. The data processing system 100 is trained (e.g., using a machine learning system) to classify polyps and tools in the images.
[0144] In some implementations, process includes receiving electronic medical records (EMR) data for the patient, the EMR data including medical information about the patient, wherein the machine learning model is trained with labeled EMR data associating values of medical information of patients with respective severity of colon disease in the patients; extracting one or more values from the EMR data to form an EMR feature vector; processing, by the machine learning model or by a second machine learning model in addition to the machine learning model, the EMR feature vector.
Re claim 4: (Original) Wang discloses the gastroscopic image diagnosis supporting system of claim 1, wherein the processor is further configured to:
generate gastroscopy entry and exit path information based on the information about the anatomical position of the video frame and a sequential position at which the video frame is acquired (e.g. the beginning of a video where an endoscope begins and a progressive movement to a certain point or boundary within the body can reflect the entry and exit information of the colon, or GI tract. In addition, the sequential positions detected within the video allow for the boundary of the scope to be align with time codes matching. This is explained in ¶ [73]-[75] and [101]-[103].); and
[0073] The landmark detection module 160 includes logic executed by the data processing system 100 for performing landmark detection within the colon. The landmark detection module 160 is configured to detect unique landmarks within the anatomical locations or regions to determine the broader region or a granular, precise location. The landmark detection module 160 is configured to detect features including one of an appendix, a cecum, an ileocecal valve, an ileum, a Taenia coli, a right colic (hepatic) flexure, a left colic (splenic) flexure, a haustra, a sigmoid flexure, a rectum, and an anus. For example, the cecum contains three possible landmarks, including an ileocecal valve, an appendiceal orifice, and Y-shaped “crow's feet” folds. Following this example, the detection of a single landmark, or combination of landmarks within an image or frame within a video, the landmark detection module 160 is able to determine that the anatomical location of this specific image or frame is the cecum.
[0074] In some cases, there are no unique landmarks present in a segment of the colon being analyzed in the endoscopy data 102. For example, the transverse and ascending colon are relatively similar and do not have unique anatomical landmarks that differentiate both segments. To assist the landmark detection module 160 to accurately predict the anatomical location without unique landmarks, the boundary detection module 162 is executed.
[0075] The boundary detection module 162 includes logic executed by the data processing system 100 for detecting boundaries in the GI tract, including a hepatic flexure, a splenic flexure, a sigmoid-descending colon junction, a rectosigmoid junction, and so forth. The boundaries are located between two segments or anatomical regions of the colon. The boundary detection module 162 is configured to determine the anatomical location of a frame in a video based on a determination if and when in the video (or corresponding image data) a particular boundary is observed. Machine learning model(s) of the boundary detection module 162 are configured to detect any visual indicators within frames within the video that correspond to these boundaries. For example, within a video, boundary detection module 162 determines that both a rectosigmoid junction and a sigmoid-descending colon junction are detected, while a splenic and a hepatic flexure have yet to be detected. Following this example, the metadata of this patient does not show that any part of the colon has been excised as part of a colectomy. As such, the boundary detection module 162 is configured to predict that the frame within the video is located in the descending colon. Such a detection is performed independently from any landmark detection conducted by the landmark detection module 160. This module overcomes the need to detect unique anatomical landmarks for each segment of the colon. However, the outputs of the landmark detection module 160 and the boundary detection module 162 can be combined to increase a confidence value that is lower with application of either individual module independently.
[0101] Coarse alignment of video of the same patient over time is performed. Insertion and withdrawal time is variable depending on the physician and findings and the length of the colon is variable depending on each patient. In order to align a recording, the algorithm will first detect boundary landmarks and record the time code (TC). These landmarks are: Cecum detected by recognizing the cecal valve, the appendix orifice or the crow path pattern of the colon; Ileum by detecting its particular colon wall pattern; a transverse colon detected by its characteristic triangular pattern. These are non-disease features.
[0102] Given the sequential nature of the video the various time stamps allows the data processing system a recognize which section is represented in the video data currently and allow the data processing system to align these landmarks over the various time points, shown in FIG. 3.
[0103] An example chronology of colon features includes the following: an Ileum; a cecum (e.g., alignment landmark is TC_cecum); an ascending colon; an end of the ascending colon and beginning of a transverse colon (e.g., alignment landmark is TC_beg_trans); an end of the transverse colon and beginning of descending colon (e.g., alignment landmark is TC_end_trans); and a rectum (e.g., align landmark is Tc_rectum). At this step the various colon section are anatomically align and time codes are matched/
provide the gastroscopy entry and exit path information to the user by displaying the gastroscopy entry and exit path information on anatomical position map through a user display (e.g. with the times of the video align and detecting the different locations within the colon, the system can display the entry and exit of the scope during an examination to a user, which is taught in ¶ [100].).
[0100] A video recorded of a full colonoscopy from insertion, (an initial time), to full withdrawal, time (an end time). These videos can be screening videos for colorectal cancer detection (CRC) or Irritable Bowel disease (IBD) videos to assess/monitor Ulcerative colitis (UC) or Crohn's diseases (CD) video. These videos will record the entire procedure, insertion of the scope will reach the cecum (CRC, UC) and may proceed to the ileum (UC, CD) and may end with a retroflexion to rectum prior to removal of the scope.
However, the combination above fails to specifically teach the feature of provide the gastroscopy entry and exit path information to the user by displaying the gastroscopy entry and exit path information on a second anatomical position map through a user display that is stomach-shaped.
However, this is well known in the art as evidenced by Kim ‘771. Similar to the primary reference, Kim ‘771 discloses displaying parts of the stomach (same field of endeavor or reasonably pertinent to the problem).
Kim ‘771 discloses provide the gastroscopy entry and exit path information to the user by displaying the gastroscopy entry and exit path information on a second anatomical position map through a user display that is stomach-shaped (e.g. the display in figure 6 shows a second anatomical position map that is also stomach-shaped. This is explained in ¶ [70]-[73] above. The second stomach-shaped image shows an entry and exit point.).
Therefore, in view of Kim ‘771, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of provide the gastroscopy entry and exit path information to the user by displaying the gastroscopy entry and exit path information on a second anatomical position map through a user display that is stomach-shaped, incorporated in the device of Wang, as modified by Park and Kim, in order to show a stomach-shaped image to a user, which allows for identification of a lesion in a specific area of the stomach more precisely (as stated in Kim ‘771 ¶ [73]).
Re claim 5: (Currently Amended) However, Wang fails to specifically teach the features of the gastroscopic image diagnosis supporting system of claim 1, wherein the processor is further configured to:
display a path of current gastroscopy and an anatomical position of a current video frame on a second anatomical position map that is stomach-shaped; and
provide the gastrointestinal anatomical position map to the user.
However, this is well known in the art as evidenced by Park. Similar to the primary reference, Park discloses a visual system of a GI tract (same field of endeavor or reasonably pertinent to the problem).
Park discloses wherein the processor is further configured to:
display a path of current gastroscopy and an anatomical position of a current video frame on a second gastrointestinal anatomical position map (e.g. for another patient and video regarding this second patient, a review of a video of the second patient’s colon can be shown associated with a digital colon mapped to the video. The digital colon can be displayed with the video of the colon of the second patient at the same time, which is taught in ¶ [190] and [191] above.); and
provide the second gastrointestinal anatomical position map to the user (e.g. the second patient’s digital colon with the video associated with the second patient is presented to the individual reviewing the colon, which is taught in ¶ [190] and [191] above.).
Therefore, in view of Park, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the processor is further configured to: display a path of current gastroscopy and an anatomical position of a current video frame on a second anatomical position map; and provide the second anatomical position map to the user, incorporated in the device of Wang, in order to display information associated with a previous examination on an GI anatomical map, which aid a physician in their treatment of a detected anomaly (as stated in Park ¶ [191]).
However, the combination above fails to specifically teach the features of display a path of current gastroscopy and an anatomical position of a current video frame on a second gastrointestinal anatomical position map that is stomach-shaped.
However, this is well known in the art as evidenced by Kim ‘771. Similar to the primary reference, Kim ‘771 discloses displaying parts of the stomach (same field of endeavor or reasonably pertinent to the problem).
Kim ‘771 discloses display a path of current gastroscopy and an anatomical position of a current video frame on a second gastrointestinal anatomical position map that is stomach-shaped (e.g. the display in figure 6 shows a second anatomical position map that is also stomach-shaped. This is explained in ¶ [70]-[73] above. The second stomach-shaped image shows an entry and exit point.).
Therefore, in view of Kim ‘771, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of display a path of current gastroscopy and an anatomical position of a current video frame on a second gastrointestinal anatomical position map that is stomach-shaped, incorporated in the device of Wang, as modified by Park and Kim, in order to show a stomach-shaped image to a user, which allows for identification of a lesion in a specific area of the stomach more precisely (as stated in Kim ‘771 ¶ [73]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kim discloses classifying anatomical positions of a stomach image.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2682
Prosecution Insights