DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-2 and 4-20 are amended. Claim 3 canceled. Claims 1-2 and 4-20 are pending in this application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2 and 4-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fink et al., US 2017/0262583 in view of Ravishankar et al., US 2025/0104451.
Regarding claim 1, Fink discloses an apparatus (fig. 2, element 212; para 0016; a computer system), comprising:
processing circuitry (fig. 2, element 216; para 0016; a processing unit) configured to:
obtain image data representing one or more medical images of a region of interest and process said image data to identify one or more abnormal portions of the image data by applying a first pre-determined model (fig. 1, element 102; para 0011; server(s) 102 (i.e., first model) may use NLP rules and dictionaries 108) to the obtained image data (fig. 4, elements 406-408; para 0031; an image of a subject's medical condition may be received (act 406) and may be analyzed, using conventional image analysis techniques, to extract image characteristics (act 408));
obtain medical text data corresponding to the one or more medical images of the region of interest and process said medical text data to identify one or more entities and associated attributes by applying a second pre-determined model (fig. 1, element 102; para 0011; server(s) 102 (i.e., second model) may use NLP rules and dictionaries 108) to the obtained medical text data (fig. 4, elements 402-404; para 0030; server(s) 102 inputting textual input 112 (act 402). Textual input 112 may include unstructured textual input including, but not limited to, doctors' notes, subject's notes, social media messages, email and text messages. Server(s) 102 may use NLP rules and dictionaries 108 to extract medical condition descriptions from textual input 112 (act 404));
perform a matching process between the one or more identified abnormal portions and the one or more identified entities to obtain matched data comprising groupings of at least one of the identified one or more abnormal portions and at least one of the identified one or more entities (fig. 4, elements 410-414; para 0031; server(s) 102 may correlate (i.e., matching) the extracted medical condition descriptions with the image characteristics (act 410) to produce a patient or subject signature (act 412). The subject signature may be compared with each of a number of reference signatures to determine one or more closest matching reference signatures (act 414). A match score may be computed. The one or more closest matching reference signatures may be determined by the match score, which may be based on computing a distance of a feature vector of the subject signature from a corresponding feature vector of each of the reference signatures. The one or more closest matching reference signatures have a minimum distance with respect to the subject signature); and
wherein the matching process is based on at least one or more properties of the identified one or more abnormal portions of the image data and at least one or more of the attributes associated with the identified one or more entities (fig. 4, elements 410-412; para 0026-0027 and 0031; server(s) 102 may correlate the extracted medical condition descriptions with the image characteristics (act 410) to produce a patient or subject signature (act 412); An example signature could be a description of a cancerous mole on the skin collocated with an image of the ailment. “Irregular shaped”, “darkened skin”, “surrounded by slightly reddish irritation area” could all appear in the text surrounding the image, along with text unrelated to the image).
Fink further discloses conventional machine learning techniques may be employed in a correlation module engine in order to correlate each extracted known medical condition description with image characteristics of a corresponding image of the known medical condition (act 310) to produce reference signatures (act 312) (fig. 3, element 312; para 0026), but Fink does not explicitly disclose store the matched data in a memory and train at least one model using the matched data and/or generate training data for the at least one model using the matched data as claimed.
However, Ravishankar discloses the image transformer 404 and the text transformer 406 can further be configured to project the respective vectors representations onto a common dimensional space 412 (i.e., storing matched data in a memory). The task-specific decoder 414 (i.e., trained model) can further learn mappings between the text prompt features as represented in the text representation vector 410 and the image features as represented in the image representation vector 408 using one or more unsupervised machine learning processes (fig. 4; para 0075-0076).
Therefore, taking the combined disclosures of Fink and Ravishankar as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the image transformer 404 and the text transformer 406 can further be configured to project the respective vectors representations onto a common dimensional space 412. The task-specific decoder 414 can further learn mappings between the text prompt features as represented in the text representation vector 410 and the image features as represented in the image representation vector 408 using one or more unsupervised machine learning processes as taught by Ravishankar into the invention of Fink for the benefit of providing an iterative framework for learning multimodal mappings tailored to medical image inferencing tasks (Ravishankar: para 0003).
Regarding claim 2, the apparatus of claim 1, Fink in the combination further disclose wherein the matching process comprises generating at least one representation of the identified one or more abnormal portions and at least one further representation of the identified one or more entities, obtaining a score representative of a degree of matching between the generated representations, and matching the one or more abnormal portions to at least one of the identified one or more entities based on the obtained score (para 0031).
Regarding claim 4, the apparatus of claim 1, Fink in the combination further disclose wherein identifying the one or more abnormal portions by the processing circuitry comprises identifying portions in comparison to a pre-determined or learned distribution for healthy and/or normal data based on a difference between a spatial distribution of the one or more medical images and a pre-determined normal distribution at a pixel or voxel level and/or as represented by a heatmap (fig. 4, element 414; para 0031).
Regarding claim 5, the apparatus of claim 1, Fink in the combination further disclose wherein processing circuitry is further configured to identify the one or more abnormal portions uses a pixel or voxel level approach, being a thresholding, a morphology, or a connected components based approach (para 0028 and 0031).
Regarding claim 6, the apparatus of claim 1, Fink in the combination further disclose wherein identifying the one or more and associated attributes by the processing circuitry comprises applying the second model to the text data to identify the one or more entities and applying the second model to obtain the attributes associated with the one or more entities (para 0022 and 0030).
Regarding claim 7, the apparatus of claim 1, Fink in the combination further disclose wherein the matching process performed by the processing circuitry comprises determining a degree of matching based on a similarity and/or a consistency between properties of the one or more abnormal portions and the attributes of the one or more entities (para 0026-0027 and 0031).
Regarding claim 8, the apparatus of claim 1, Fink in the combination further disclose wherein the matching process performed by the processing circuitry comprises determining a similarity function or other measure of distance between mathematical representations of the one or more properties of the identified one or more abnormal portions and the attributes of the identified one or more entities (para 0026-0027 and 0031-0032).
Regarding claim 9, the apparatus of claim 1, Fink in the combination further disclose wherein the matching process performed by the processing circuitry is based on a pre-determined relationships between the one or more properties and the one or more attributes (para 0026-0027 and 0031).
Regarding claim 10, the apparatus of claim 1, Fink in the combination further disclose wherein the matching process performed by the processing circuitry is based on minimizing or otherwise optimizing a matching function, the matching function comprising a term representing a similarity between the identified one or more abnormal portions and/or a term penalizing a variation of those abnormal image portions assigned to a same class of entities (para 0031).
Regarding claim 11, the apparatus of claim 1, Fink in the combination further disclose wherein the matching process comprises performing an optimization process being a Jonker-Volegenant algorithm applied to solve multiple linear assignment problems (para 0031).
Regarding claim 12, the apparatus of claim 1, Fink in the combination further disclose wherein the processing circuitry is further configured to retrain and/or refine the first model and/or the second model using the obtained matched data (fig. 3; para 0022-0029).
Regarding claim 13, the apparatus of claim 1, Fink in the combination further disclose wherein the processing circuitry is further configured to display the matched data including the groupings to a user, and obtain further user input representing a user evaluation of the matched data as part of a further training process or as part of the generation of training data (fig. 4, element 416; para 0033-0034).
Regarding claim 14, the apparatus of claim 1, Fink in the combination further disclose wherein the first model and/or the second model comprises a deep learning or other artificial neural network based model (para 0026).
Regarding claim 15, the apparatus of claim 1, Fink in the combination further disclose wherein the processing circuitry is further configured to apply a principle component analysis or other feature reduction procedure to a larger set of features, and wherein at least part of the matching process is applied to the reduced set of features (fig. 4, element 410; para 0031).
Regarding claim 16, the apparatus of claim 1, Fink in the combination further disclose wherein the image data comprises 1D, 2D, 3D, or 4D data, and/or wherein the image data (fig. 1, element 106; para 0022 and 0025) comprises at least one of: CT, MRI, fluoroscopy, ultrasound data, or medical imaging data obtaining using an other modality; ECG data or other medical measurement data; volumetric data or slice data; or time series data (fig. 1, element 106; para 0022 and 0025).
Regarding claim 17, the apparatus of claim 1, Fink in the combination further disclose wherein the one or more properties of the identified one or more abnormal portions comprise at least one of: an intensity, a texture, a shape, a location, and a measure of abnormality of at least the one or more abnormal portions (para 0027-0029).
Regarding claim 18, the apparatus of claim 1, Fink in the combination further disclose wherein an entity of the one or more entities comprises at least one of a finding, an impression, or other observable, wherein the entity is associated with a pathology (para 0023-0024), and wherein the one or more attributes associated with the entity may comprise attributes associated with an anatomical location or region, an anatomical distribution, laterality, severity, or a level of certainty (para 0023-0024).
Regarding claim 19, the apparatus of claim 1, Fink in the combination further disclose wherein the matching process is further based on further information obtained from the medical text data, including author information, a measure of quality, content of the medical text data, and/or other metadata (para 0022 and 0030).
Regarding claim 20, this claim recites substantially the same limitations that are performed by claim 1 above, and it is rejected for the same reasons.
Response to Arguments
Applicant's arguments with respect to claims 1-2 and 4-20 have been considered but are moot in view of the new ground(s) of rejection.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAN D HUYNH whose telephone number is (571)270-1937. The examiner can normally be reached 8AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAN D HUYNH/Primary Examiner, Art Unit 2665
Prosecution Insights