DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Application Status
This is the first non-final action on the merits. Claims 1-18 as originally filed on July 11, 2024 are currently pending and considered below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on December 13, 2024 is being considered by the examiner. The submission is in compliance with the provisions of 37 CFR 1.97.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without significantly more.
Claims 1-6 recite a method for automatically generating a structured medical record from endoscopy data, which is within the statutory category of a process. Claims 7-12 recite a data processing system configured for automatically generating a structured medical record from endoscopy data, which is within the statutory category of a machine. Claims 13-18 recite one or more non-transitory computer readable media storing instructions for automatically generating a structured medical record from endoscopy data, which is within the statutory category of an article of manufacture.
Step 2A - Prong One:
Regarding Prong One of Step 2A, the claim limitations are to be analyzed to determine whether, under their broadest reasonable interpretation, they "recite" a judicial exception or in other words whether a judicial exception is "set forth" or "described" in the claims. An "abstract idea" judicial exception is subject matter that falls within at least one of the following groupings: a) mathematical concepts, b) certain methods of organizing human activity, and/or c) mental processes. Representative independent claim 1 includes limitations that recite at least one abstract idea.
Specifically, independent claim 1 recites: A method for automatically generating a structured medical record from endoscopy data, the method comprising:
obtaining image data including endoscopic images of a gastrointestinal (GI) tract of a patient;
determining one or more features to extract from the image data, the features each representing a physical parameter of the GI tract;
extracting the one or more features from the image data;
processing the features to generate transformed features corresponding to fields of a structured medical record; and
storing, in a data store, one or more data entries including the transformed features, wherein the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the transformed features in response to receiving the structured queries.
The underlined limitations are directed to methods of organizing human activity. The claim recites steps of obtaining image data, determining features to extract from the image data, extracting the features, processing the features, storing data entries, receiving structured queries for the data entries and providing the data entries. These steps, under its broadest reasonable interpretation, are categorized as methods of organizing human activity, specifically associated with managing personal behavior or relationships or interactions between people (e.g. creating a medical record). The claim encompasses a person following rules or instructions to receive and process data in the manner described in the abstract idea. If the claim limitation, under its broadest reasonable interpretation, covers managing personal behavior or interactions between people but for the recitation of generic computer components, then it falls within the “Certain Methods of Organizing Human Activity” grouping of abstract ideas. See MPEP § 2106.04(a). The Examiner further notes that “Certain Methods of Organizing Human Activity” includes a person's interaction with a computer (see October 2019 Update: Subject Matter Eligibility at Pg. 5). The abstract idea for Claims 7 and 13 are identical as the abstract idea for Claim 1, because the only difference between Claim 1 and 7 is that Claim 1 recites a method, whereas Claim 7 recites a system, and because the only difference between Claims 1 and 13 is that Claim 1 recites a method, whereas Claim 13 recites one or more non-transitory computer readable media. Any limitation not identified above as part of methods of organizing human activity, are deemed “additional elements” and will be discussed further in detail below. Accordingly, claims 1 and 13 recite at least one abstract idea.
Similarly, dependent claims 2-6, 8-12 and 14-18 further narrow the abstract idea described in the independent claims. Claims 2-4, 8-10 and 14-16 describe applying a model to the image data. Claims 5, 10 and 16 describe the structured medical record. Claims 6, 11 and 17 describe the structured queries. Claims 2, 8 and 14 partially narrow the abstract idea as described above, and also introduce additional element(s) which will be discussed in Step 2A Prong 2 and Step 2B. These limitations only serve to further limit the abstract idea and hence, are directed toward fundamentally the same abstract ideas as independent claims 1, 7 and 13, even when considered individually and as an ordered combination.
Step 2A - Prong Two:
Regarding Prong Two of Step 2A, it must be determined whether the claim as a whole integrates the abstract idea into a practical application. It must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a "practical application."
In the present case, claims 1-18 as a whole do not integrate the abstract idea into a practical application because they do not impose meaningful limits on practicing the abstract idea. The additional elements or combination of additional elements, beyond the above-noted at least one abstract idea will be described as follows (where the bolded portions are the “additional limitations” while the underlined portions continue to represent the “abstract idea(s)”).
Specifically, independent claim 1 recites: A method for automatically generating a structured medical record from endoscopy data, the method comprising:
obtaining image data including endoscopic images of a gastrointestinal (GI) tract of a patient;
determining one or more features to extract from the image data, the features each representing a physical parameter of the GI tract;
extracting the one or more features from the image data;
processing the features to generate transformed features corresponding to fields of a structured medical record; and
storing, in a data store, one or more data entries including the transformed features, wherein the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the transformed features in response to receiving the structured queries.
The independent claims recite the additional elements of a data store, data processing system, processor, and memory that implement the identified abstract idea. The data store, data processing system, processor, and memory are not described by the applicant and are recited at a high-level of generality such that they amounts to no more than mere instructions to apply the exception using a generic computer component (i.e., merely invoking the computer structure as a tool used to execute the limitations, MPEP 2106.05(f)).
The dependent claims 2, 8 and 14 recite additional element(s) beyond those already recited in the independent claims that implement the identified abstract idea. Claims 2, 8, and 14 recite a machine learning model. However, these additional elements do not integrate the abstract idea into a practical application because, as stated above, they represent mere instructions to apply the abstract idea on a computer (i.e., merely invoking the machine learning model as a tool used to execute the limitations).
Accordingly, the claims as a whole do not integrate the abstract idea into a practical application as they do not impose any meaningful limits on practicing the abstract idea.
Step 2B
Regarding Step 2B, representative independent claim 1 does not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application.
When viewed as a whole, claims 1-18 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims recite processes that are abstract and simply implementing the process on a computer(s) is not enough to qualify as "significantly more."
As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of using a data store, data processing system, processor, and memory to perform the noted steps amount to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept (“significantly more”).
The dependent claims 2, 8 and 14 recite additional element(s) beyond those already recited in the independent claims that implement the identified abstract idea. Claims 2, 8, and 14 recite machine learning. However, these functions are not deemed significantly more than the abstract idea because, as stated above, they represent mere instructions to apply the abstract idea on a computer (i.e., merely invoking the computer structure as a tool used to execute the limitations).
Therefore, claims 1-18 are rejected under 35 USC §101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7-11 and 13-17 are rejected under 35 U.S.C. 103 as being unpatentable over Mohapatra (US 2022/0398458 A1) in further view of Li (WO 2020/233254 A1) and Kim (WO 2024/083311 A1).
Regarding claim 1, Mohapatra teaches: A method for automatically generating a structured medical record from endoscopy data (e.g. see [0001]-[0002]), the method comprising:
obtaining image data including endoscopic images of a gastrointestinal (GI) tract of a patient; (“The endoscope…can initiate capture of a video of a colonoscopy procedure…an image of a polyp, or other videos and/or images associated with the videos and/or images”, e.g. see [0072])
determining one or more features to extract from the image data, the features each representing a physical parameter of the GI tract; extracting the one or more features from the image data; (determining image-based features including “color, vessels, and surface pattern”, e.g. see [0025]; “extract frames 727 from the video, or images of the videos and/or images 724 associated with the colonoscopy procedure”, e.g. see [0068]; extraction of specific physical parameters from these images, such as color, vascular patterns, surface patterns and 3D DNA shape, e.g. see [0038]; “Image-based features of the videos and/or images 724 can include a numerical feature.”, e.g. see [0063]; combining “clinical information and the image-based features” to form a “feature pool (e.g., of the model data 730)”, e.g. see [0064])
Mohapatra does not teach:
processing the features to generate transformed features corresponding to fields of a structured medical record;
storing, in a data store, one or more data entries including the transformed features […]
However, Li in the analogous art of medical image analysis (e.g. see [0005]) teaches:
processing the features to generate transformed features corresponding to fields of a structured medical record; (“calculating an area Si, a center point coordinate (xi, yi), a transverse longest distance Wi, and a longitudinal longest distance Hi of each target region Ri, and acquiring an orientation number Pi of the target region Ri, and associating Si, (x, y), Wi, Hi, Pi as a feature and a patient id to form an image information structured data table” (This table acts as a structured record where visual features are transformed into specific data fields.), e.g. see [0039])
storing, in a data store, one or more data entries including the transformed features […] (a “distributed data analysis platform” composed of servers implemented by “deploying Hadoop and Spark clusters” to store the “structured data table”, e.g. see [0057], [0049])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra to include processing the features to generate transformed features corresponding to fields of a structured medical record and storing, in a data store, one or more data entries including the transformed features as taught by Li, for the purposes of “structur[ing] and transform[ing]” key image information so it can be fused with other patient data to form a system with “high efficiency and high utilization” (Li [0027]).
Mohapatra and Li do not teach:
wherein the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the features in response to receiving the structured queries.
However, Kim in the analogous art of medical image analysis (e.g. see pg. 4 para. 3) teaches:
wherein the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the features in response to receiving the structured queries. (a system for “providing medical reference image data” that performs “database query”, e.g. see abstract; the search uses as “input parameters” information retrievable from the report and “identifies in the database subsets…to provide them as output medical image data”, e.g. see pgs 5-6 para. 5)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra and Lin to include the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the features in response to receiving the structured queries as taught by Kim, for the purposes of “better integrat[ing] these operations into the physician's workflow” as existing search tools are “decoupled from the core workflow of radiology reporting”(Kim, pg. 5 para. 3, pg. 2 para. 1).
Regarding claim 2, Mohapatra, Li and Kim teach the method of claim 1 as described above.
Mohapatra further teaches:
applying a machine learning model to the image data, the machine learning model being trained with images of GI tracts; (applying a “deep learning” system to “video or images associated with the colonoscopy”, abstract; the “deep structured learning algorithm” is trained on images to detect colorectal polyps, e.g. see [0022], [0030])
identifying, from the applying, a malignancy in the GI tract; (identifying and classifying polyps into “malignant”, e.g. categories such as “adenoma”, e.g. see [0071], [0067])
determining one or more physical features of the malignancy; […] (determining image-based features, including numerical features, of the polyps such as “color, vessels, and surface pattern”, e.g. see [0025], [0063]).
Mohapatra does not teach:
outputting the one or more physical features as feature data
However, Li in the analogous art teaches:
outputting the one or more physical features as feature data (calculating discrete data points, e.g. Si, (x, y), Wi, Hi, Pi, and outputting them into a “structured data table”, e.g. see [0039])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra to include outputting the one or more physical features as feature data as taught by Li, for the purposes of “structur[ing] and transform[ing]” key image information so it can be fused with other patient data to form a system with “high efficiency and high utilization” (Li [0027]).
Regarding claim 3, Mohapatra, Li and Kim teach the method of claim 2 as described above.
Mohapatra further teaches:
wherein the one or more physical features comprise a color of the malignancy, an outer shape of the malignancy, an area of the malignancy, a location of the malignancy, or an orientation of the malignancy (analyzing the color of the adenoma polyp and its location withing the frame, e.g. see [0044]; “Naming color” as a specific parameter for classification of the malignancy, e.g. see [0038]; assessing the structural form of the lesion including “surface patterns”, e.g. oval, tubular, or branched structures, and “3D DNA Shape”, e.g. see [0025], [0038]).
Regarding claim 4, Mohapatra, Li and Kim teach the method of claim 2 as described above.
Mohapatra does not teach:
comparing the one or more physical features to corresponding one or more features extracted from other image data; and generating comparative features data
However, Li in the analogous art teaches:
comparing the one or more physical features to corresponding one or more features extracted from other image data; and generating comparative features data (analyzing “different CT image analysis results of the same patient id”; the system extracts the area S1 of the feature from the first image and the area S2 from the corresponding feature of the second image and compares these features by evaluating them to find the “largest Si” (i.e. comparative features data), which is used to form the “structured information data”, e.g. see [0067], [0061]-[0063])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra to include comparing the one or more physical features to corresponding one or more features extracted from other image data and generating comparative features data as taught by Li, for the purposes of attaining “rapid specificity analysis processing capability” required of medical image data (Li [0002]).
Regarding claim 5, Mohapatra, Li and Kim teach the method of claim 1 as described above.
Mohapatra does not teach:
wherein the structured medical record comprises a set of instances of records each associated with a common identifier, and wherein the method comprises:
comparing data of a first field of a first instance of the instances of the records with a second field of a second instance of the instances of the records for generating comparative feature data
However, Li in the analogous art teaches:
wherein the structured medical record comprises a set of instances of records each associated with a common identifier, and wherein the method comprises: (forming an “image information structured data table” (i.e. structured medical record) that stores multiple different “CT image analysis results” linked to the “same patient id” (i.e. a common identifier), e.g. see [0039], [0067])
comparing data of a first field of a first instance of the instances of the records with a second field of a second instance of the instances of the records for generating comparative feature data (analyzing “different CT image analysis results of the same patient id”; the system extracts the area S1 of the feature from the first image (i.e. a first field of a first instance) and the area S2 from the corresponding feature of the second image (i.e. a second field of a second instance) and compares these features by evaluating them to find the “largest Si” (i.e. comparative features data), which is used to form the “structured information data”, e.g. see [0067], [0061]-[0063])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra to include the structured medical record comprises a set of instances of records each associated with a common identifier, and comparing data of a first field of a first instance of the instances of the records with a second field of a second instance of the instances of the records for generating comparative feature data as taught by Li, for the purposes of attaining “rapid specificity analysis processing capability” required of medical image data (Li [0002]).
Regarding claim 7, Mohapatra teaches: A data processing system configured for automatically generating a structured medical record from endoscopy data (e.g. see [0001]-[0002]), the data processing system comprising:
at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: (e.g. see [0059])
obtaining image data including endoscopic images of a gastrointestinal (GI) tract of a patient; (“The endoscope…can initiate capture of a video of a colonoscopy procedure…an image of a polyp, or other videos and/or images associated with the videos and/or images”, e.g. see [0072])
determining one or more features to extract from the image data, the features each representing a physical parameter of the GI tract; extracting the one or more features from the image data; (determining image-based features including “color, vessels, and surface pattern”, e.g. see [0025]; “extract frames 727 from the video, or images of the videos and/or images 724 associated with the colonoscopy procedure”, e.g. see [0068]; extraction of specific physical parameters from these images, such as color, vascular patterns, surface patterns and 3D DNA shape, e.g. see [0038]; “Image-based features of the videos and/or images 724 can include a numerical feature.”, e.g. see [0063]; combining “clinical information and the image-based features” to form a “feature pool (e.g., of the model data 730)”, e.g. see [0064])
Mohapatra does not teach:
processing the features to generate transformed features corresponding to fields of a structured medical record;
storing, in a data store, one or more data entries including the transformed features […]
However, Li in the analogous art teaches:
processing the features to generate transformed features corresponding to fields of a structured medical record; (“calculating an area Si, a center point coordinate (xi, yi), a transverse longest distance Wi, and a longitudinal longest distance Hi of each target region Ri, and acquiring an orientation number Pi of the target region Ri, and associating Si, (x, y), Wi, Hi, Pi as a feature and a patient id to form an image information structured data table” (This table acts as a structured record where visual features are transformed into specific data fields.), e.g. see [0039])
storing, in a data store, one or more data entries including the transformed features […] (a “distributed data analysis platform” composed of servers implemented by “deploying Hadoop and Spark clusters” to store the “structured data table”, e.g. see [0057], [0049])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra to include processing the features to generate transformed features corresponding to fields of a structured medical record and storing, in a data store, one or more data entries including the transformed features as taught by Li, for the purposes of “structur[ing] and transform[ing]” key image information so it can be fused with other patient data to form a system with “high efficiency and high utilization” (Li [0027]).
Mohapatra and Li do not teach:
wherein the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the features in response to receiving the structured queries.
However, Kim in the analogous art teaches:
wherein the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the features in response to receiving the structured queries. (a system for “providing medical reference image data” that performs “database query”, e.g. see abstract; the search uses as “input parameters” information retrievable from the report and “identifies in the database subsets…to provide them as output medical image data”, e.g. see pgs 5-6 para. 5)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra and Lin to include the data store is configured to receive structured queries for the data entries in the data store and provide the data entries including the features in response to receiving the structured queries as taught by Kim, for the purposes of “better integrat[ing] these operations into the physician's workflow” as existing search tools are “decoupled from the core workflow of radiology reporting”(Kim, pg. 5 para. 3, pg. 2 para. 1).
Claims 8 and 14 recite substantially similar limitations as those already addressed in claim 2, and, as such are rejected for similar reasons as given above.
Claims 9 and 15 recite substantially similar limitations as those already addressed in claim 3, and, as such are rejected for similar reasons as given above.
Claims 10 and 16 recite substantially similar limitations as those already addressed in claim 4, and, as such are rejected for similar reasons as given above.
Claims 11 and 17 recite substantially similar limitations as those already addressed in claim 5, and, as such are rejected for similar reasons as given above.
Claim 13 recites substantially similar limitations as those already addressed in claim 7, and, as such is rejected for similar reasons as given above.
Claims 6, 12 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Mohapatra, Li and Kim in further view of Ullaskrishnan (US 2024/00170112 A1).
Regarding claim 6, Mohapatra, Li and Kim teach the method of claim 1 as described above.
Mohapatra, Li and Kim do not teach:
wherein the structured queries comprise a natural language query, and
wherein the method comprises processing the natural language query using a language model trained based on a library of terms associated with the fields of the structured medical record
However, Ullaskrishnan in the analogous art of “field of data retrieval based on a query…from an electronic medical record database” (e.g. see [0002]) teaches:
wherein the structured queries comprise a natural language query, and (“the query may comprise natural language…receiving the query may comprise obtaining an utterance from a user… receiving the query may comprise obtaining a text input by a user”, e.g. see [0027])
wherein the method comprises processing the natural language query using a language model trained based on a library of terms associated with the fields of the structured medical record (“providing a natural language processing algorithm” which “comprises a transformer network”, e.g. see [0068], [0076]; “the natural language processing algorithm may comprise a large language model”, e.g. see [0083]; “the fine-tuning may comprise further training the transformer network with medical texts with…medical ontologies such as RadLex and/or SNOMED” (i.e. library of terms); “SNOMED…is a systematically organized computer-processable collection of medical terms”, e.g. see [0078], [0046]; “the medical ontology…encodes references of codes of the medical ontology to data identifiers of the electronic medical record database”, e.g. see [0152]; “sub-unit 23 may be configured to match the codes…with data identifiers of the electronic medical record database EMR and query the EMR on that basis”, e.g. see [0188])
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Mohapatra, Li and Kim to include the structured queries comprise a natural language query and processing the natural language query using a language model trained based on a library of terms associated with the fields of the structured medical record as taught by Ullaskrishnan, for the purposes of “allowing for an efficient provision of information from an electronic medical record database based on a query containing natural language” (Ullaskrishnan [0007]).
Claims 12 and 18 recite substantially similar limitations as those already addressed in claim 6, and, as such are rejected for similar reasons as given above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Reference Tso (US 2024/0006037 A1) discloses simultaneous generation of electronic medical records for artificial intelligence-enabled dynamic image recognition. Reference McKinney (US 2021/0065859 A1) discloses automated extraction of structured labels from medical text using deep convolutional networks. Reference Reicher (US 2018/0060488 A1) discloses customizing annotations on medical images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaisha Abdullah whose telephone number is (571)272-5668. The examiner can normally be reached Monday through Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Peter Choi can be reached on (469) 295-9171. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.A./
/PETER H CHOI/Supervisory Patent Examiner, Art Unit 3681