DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims and Rejections
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Claims 1, 4, 6, 8, 10, 11, 14, 15, 17, 18, and 20 have been amended.
Claims 1-20 are currently pending.
In light of the claim amendments, the Section 101 rejection has been withdrawn.
Claim Objections
Claims 1-20 are objected to because of the following informalities:
Claims 1-20 change between detecting “signs of [the] infection” (see e.g., claims 1, 5, 6, 7, 8, 10, 12-15, 17, 19, 20) to, more directly, “detect [the] infection” (see, e.g., claims 1-3, 8, 15). Claims 1-20 also change between detecting one “infection” (see, e.g., claims 6, 7, 10, 12, 13, 14, 17, 19, 20) to multiple “infections” (see e.g., claims 1, 2, 3, 8, 15). To be clearer and more consistent, the claims should be amended to recite detecting “signs of infection” or “signs of the infection.”
Claims 2-7 recite a computing device but then recite method steps without reference to the one or more processors performing those steps. Claims 2-7 should be amended to read “The computing device for detecting ear infections of claim 1, wherein the operations are further comprising:….”
Claims 2 and 3 should be amended to more simply recite “and wherein the artificial intelligence model is the trained model.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Each of claims 1, 8, and 15 recite or similarly recite: “generating, from the sensor data, an occlusion-compensated thermal feature vector including at least grid-coordinate temperature values, a coverage metric of the tympanic membrane by the sensor array, and heat signature data derived from the temperature information.” It would not be clear to one having ordinary skill in the art what is meant by “feature vector.” Applicant’s disclosure does not use the term nor does Applicant describe what is typically meant by the term feature vector. In at least one possible meaning, as understood by the Examiner, a feature vector can be an array of measurable properties or features that are combined as a single input for an ML model. For example, while describing machine-learning methods, DOUGLAS teaches that such methods “typically work on a fixed set of features as inputs to return a response, or output. A set of features consists of scalar values that summarize the input (e.g., an image) and can be easily processed with a computer…These features together comprise a ‘feature vector’ that would be extracted from the image before being fed as input into a machine learning method to predict some response (e.g., diagnosis, prognosis, etc.).” ([0210]; see also [0229]-[0230] describing how features are combined into a feature vector).
However, Applicant’s disclosure does not describe generating or inputting a feature vector so it is not clear what Applicant’s intended meaning is. Applicant’s disclosure describes different inputs and using those inputs to train the machine-learning model, but Applicant does not describe generating a “feature vector” that includes multiple features combined as a single input (e.g., array). For example, Applicant describes using the following as inputs: (1) sensor data/temperature data (e.g., grid-coordinate data) ([0105], [0176]); (2) heat signature patterns, temperature gradients or distributions ([0169], [0170], [0181]); (3) temperature differentials ([0108]); (3) spatial patterns ([0108]); and (4) historical comparisons ([0108], [0122]). However, the disclosure does not teach combining these features into a common feature vector that is input into a machine-learning model.
All dependent claims either depend directly or indirectly from claim 1, claim 8, or claim 15. As such, all claims are indefinite based on their dependency.
For the above reasons, claims 1-20 are indefinite.
For the purpose of a compact prosecution, Examiner is interpreting the relevant limitation as follows:
generating, from the sensor data, inputs that at least include grid-coordinate temperature values, a coverage metric of the tympanic membrane by the sensor array, and heat signature data derived from the temperature information, wherein at least one of the inputs has been compensated for the presence of occlusions;
applying the the inputs to a trained model configured to detect infections in the ear canal of the patient; and
detecting, based at least in part on the inputs and the trained model, signs of infection in the ear canal of the patient.
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
“To satisfy the written description requirement, a patent specification must describe the claimed invention in sufficient detail that one skilled in the art can reasonably conclude that the inventor had possession of the claimed invention.” (MPEP 2163, I).
As explained above in the Section 112(b) rejection, Applicant’s disclosure does not adequately support generating a “feature vector” that is based on the inputs recited in the claimed invention. While Applicant’s disclosure describes using different inputs to detect signs of infection, it does not describe generating a feature vector with each of the recited inputs or generating a feature vector in general.
Moreover, the term “coverage metric,” as an input to a machine-learning model, lacks adequate written description support. First, the term “coverage metric” is not used in Applicant’s disclosure. Second, Applicant’s disclosure does not use a term that is synonymous with or similar to coverage metric or describe an example that might be considered a coverage metric. The term “metric” implies a quantifiable measurement or a calculation based on measurements. (see, e.g., Oxford English Dictionary, “metric (n.1 & adj.1),” June 2026, https://doi.org/10.1093/OED/1040092176 or “Metric.” Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/metric. Accessed 24 Sep. 2026). However, Applicant does not describe inputting a value that represents some form of coverage. The closest teaching to “coverage” in Applicant’s disclosure includes “providing a visual representation of the sensor array’s coverage within the patient’s ear canal” to enhance diagnostic clarity. However, a visual representation is not a quantifiable measurement or calculated value. More importantly, this description of the visual representation was provided in the context of guiding the user to obtain a better image. It was not described as an input to a machine-learning model. Likewise, other uses of the term “coverage” are not used in the context of model inputs.
All dependent claims either depend directly or indirectly from claim 1, claim 8, or claim 15. As such, all claims lack adequate written description support.
For the above reasons, claims 1-20 are rejected for lacking written description support.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-9, 12, 15, 16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2015/0351637 A1 (hereinafter “RUPPERSBERG”) in view of U.S. Patent Appl. Publ. No. 2015/0065803 A1 (hereinafter “DOUGLAS”) and U.S. Patent Appl. Publ. No. 2024/0366145 A1 (hereinafter “DACOSTA”).
With respect to claim 1 (and in light of the Section 112(b) rejection), RUPPERSBERG teaches a computing device for detecting ear infections (Title and Abstract). Although described in the context of determine a body core temperature, the RUPPERSBERG is also described as being capable of “diagnosing an ear disease,” which may include “reliably diagnosing e.g. an inflammation of the eardrum without the need of assistance of a skilled physician.” ([0133]). The device comprising:
one or more processors. Claim 3 recites “a logic unit configured for receiving and processing signals from the infrared sensor unit….”.
a sensor array configured to measure tympanic temperatures across multiple points on a tympanic membrane of a patient. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). RUPPERSBERG also describes technology enabling thermal images of sufficient resolution. “Therefore, wafer-level imaging technology allows obtaining images (of the temperature distribution and/or of light in the visual range) of “sufficient” resolution of the eardrum, e.g. images of 250 pixels×250 pixels, with a footprint of the camera (including a lens) of only about 1 mm×1 mm or even smaller.” ([0052]).
a housing configured to contain the one or more processors and the sensor array, the housing having a distal end configured for at least partial insertion into an ear canal of the patient. Figure 1 of RUPPERSBERG shows an ear inspection device 10 having a head portion 14 that includes a logic unit 44. The head portion 14 includes a distal end 18 that “is adapted to be introduced into a subject's ear canal.” ([0155]).
receiving an indication that a portion of the device has been inserted into an ear canal of the patient. RUPPERSBERG describes a “moving mechanism” that is triggered when the device is inserted into the ear canal. “As soon as the probe cover 60 gets in contact with an inner lateral surface of the ear canal, a friction force is exerted on the probe cover 60. The friction force depends on the position of the head portion 14 within the ear canal: the friction force increases with increasing insertion depth.” ([0193]). The moving mechanism is electrically connected to one of the cameras. “In case the probe cover 60 is axially displaced, the motion detector can emit an electric signal which is transmitted to the at least one camera 40.1 or any logical unit or control unit, evoking start-up or powering of the camera 40.1. In such a way, by means of motion detection or detection of the axial position of the probe cover 60, the camera 40.1 can be powered at a time when the camera 40.1 is in visual communication with the eardrum.” ([0199]). Notably, the imaging unit and infrared sensor unit operate simultaneously. ([0165]).
receiving sensor data from the sensor array, the sensor data indicating temperature information at grid-coordinate locations of the sensor array. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]).
RUPPERSBERG does not explicitly teach that the device includes a non-transitory computer-readable media, housed in the housing, storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. Nonetheless, RUPPERSBERG a logic unit 44 that processes infrared/imaging data and is configured to evaluate the images. (see, e.g., claims 3 and 4).
DOUGLAS, on the other hand, teaches “methods and apparatuses for assisting in the acquisition and analysis of images of the tympanic membrane to provide information that may be helpful in the understanding and management of disease, such ear infection (acute otitis media).” (Abstract). DOUGLAS teaches various steps (described in greater detail below) for analyzing images of the tympanic membrane to detect signs of infection. DOUGLAS further teaches that these steps may be carried out by a processor that executes instructions stored on a computer-readable medium. “For example, an apparatus may also include an otoscope or otoscope adapter (lens portion) to connected to a mobile telecommunciations device and a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor, that when executed by the processor causes the processor to….” ([0031]; see also [0032] and [0051]).
It would have been obvious to one having ordinary skill in the art at the time of filing to add a non-transitory computer-readable media to the RUPPERSBERG device, housed in the housing and storing instructions, as recited. One of ordinary skill in the art could have added a non-transitory computer-readable medium within the housing using known methods. One of ordinary skill in the art would have recognized that the results of adding the computer-readable medium would be predictable.
RUPPERSBERG does not explicitly teach generating, from the sensor data, an occlusion-compensated thermal feature vector including at least grid-coordinate temperature values, a coverage metric of the tympanic membrane by the sensor array, and heat signature data derived from the temperature information; applying the occlusion-compensated thermal feature vector to a trained model configured to detect infections in the ear canal of the patient; and detecting, based at least in part on the occlusion-compensated thermal feature vector and the trained model, signs of infection in the ear canal of the patient. Nonetheless, RUPPERSBERG:
teaches that ear infections can be identified based on temperature information. (see, e.g., [0028]: “Local inflammations also lead to a raise in temperature at the site of inflammation” and [0068]: “Inflammation of the eardrum may suggest e.g. an (bacterial/viral) infection”).
teaches that it is critical that the analysis be based on image data of the tympanic membrane and not, for example, the canal walls or obstructions (e.g. ear wax). ([0003]; see also [0009]: “In particular, it is an object of the present invention to provide an ear inspection device or a method of acquiring signals or data that allow for reliable differentiation of the eardrum.”; and [0021]: “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]).
teaches that the unwanted objects have a different temperature than the eardrum. “[T]he temperature at the surface of the eardrum (substantially corresponding to the subject's body core temperature) is higher than the temperature of the wall of the exterior ear canal and/or of earwax, hair or dirt in the exterior ear canal.” ([0021]).
teaches removing or disregarding objects that are not of interest to improve analysis. “In order to improve the identification of objects in the subject's ear, the logic unit is preferably further configured for identifying and discriminating different objects in the subject's ear, such as earwax, hair and the eardrum….” ([0031]). “[O]bjects shown in the at least on captured image may be identified (and distinguished from other objects in the subject's ear), and then the status (especially the temperature) of at least one of the identified objects is determined.” ([0133]).
DOUGLAS teaches extracting features and determining the probability those features identify the tympanic membrane. ([0014]). DOUGLAS teaches generating feature vectors derived from images of tympanic membranes and inputting the feature vectors into a trained model. ([0053]-[0054]). “Machine learning methods typically work on a fixed set of features as inputs to return a response, or output. A set of features consists of scalar values that summarize the input (e.g., an image) and can be easily processed with a computer…These features together comprise a ‘feature vector’ that would be extracted from the image before being fed as input into a machine learning method to predict some response (e.g., diagnosis, prognosis, etc.).” ([0210]; see also [0229]-[0230] describing how features are combined into a feature vector).
DOUGLAS further teaches that the features must be carefully selected and extracted. (see, e.g., [0211]). “[I]mage features are typically extracted via a feature extraction method that is resistant to translation, scaling and other deformations and is designed specifically with the image subject in mind.” ([0212]). DOUGLAS then describes various features that could be used with images of tympanic membranes. DOUGLAS even describes temperature as one example of “numerical information” that can be represented by scalar, real-numbered values that are “suitable for directly being used as features in a machine learning system….”
Similar to RUPPERSBERG, DOUGLAS is concerned with a sufficient amount of the tympanic membrane being visible and, as such, teaches guiding the user so that the tympanic membrane is sufficiently viewable. “In general, a method of guidance or an apparatus for guiding a subject to take an image may examine images (digital images) of a patient's ear canal being taken by the user, e.g., operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%,….” ([0036]). DOUGLAS is also concerned with obstructions. “Obstructions within the ear canal, including cerumen or foreign bodies may either prevent the ear exam from being completed successfully (because they partially or completely occlude the tympanic membrane) or may present a hazard during the otoscopic examination because they may be pushed by the speculum deeper into the ear.” ([0328]).
DOUGLAS teaches that portions of the image that do not include the eardrum should be excluded. First, an image is analyzed to identify those subregions that are likely part of the tympanic membrane. (see, e.g., [0014]-[0015]). When generating the feature vector, the portions that are not part of the tympanic membrane can be excluded. “The image is divided into blocks of a certain size (example 32×32) and regions that have certain characteristics that do not imply the presence of eardrum could be excluded automatically.” ([0248]). DOUGLAS further teaches that obstructions could be identified in a similar manner as to how regions of the tympanic membrane are identified. ([0328]-[0329]).
Accordingly, DOUGLAS teaches generating, from the sensor data, an occlusion-compensated thermal feature vector. DOUGLAS also teaches applying the occlusion-compensated thermal feature vector to a trained model configured to detect infections in the ear canal of the patient (see, e.g., [0256], see also [0252]-[0254]) and detecting, based at least in part on the occlusion-compensated thermal feature vector and the trained model, signs of infection in the ear canal of the patient. (see, e.g., [0256], see also [0241]-[0242]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG computing device to generate a feature vector that includes sensor data (i.e., grid-coordinate temperature values) and that was compensated for to remove unnecessary information. One would be motivated to use a feature vector as an input to machine-learning model as ML models are capable of detecting signs of ear infection based on image data. Moreover, both RUPPERSBERG and DOUGLAS teach identifying and excluding unwanted image data. Prior to generating the feature vector the grid-coordinates associated with canal walls and earwax or other objects would be excluded as taught by DOUGLAS. There would have been a reasonable expectation of success as DOUGLAS teaches excluding regions that do not correspond to the tympanic membrane.
Moreover, it would have been obvious to one having ordinary skill in the art at the time of filing to modify the feature vector to include a coverage metric. DOUGLAS teaches that one diagnostically relevant piece of information is the amount of the tympanic membrane that is captured within the image. After excluding unwanted data associated with the canal wall and obstructions, one having ordinary skill in the art would include, with the feature vector, a coverage metric representing how much of the tympanic membrane is represented by the feature vector. For example, if the image field of view and obstructions only reveal 50% of the tympanic membrane, then the feature vector will include a coverage metric to indicate that the grid-coordinate temperature values do not represent the entire tympanic membrane. As the percentage of the tympanic membrane decreases, the confidence score in the output of the ML model would necessarily decrease. There would have been a reasonable expectation of success as DOUGLAS teaches that feature vectors are a combination of relevant scalar values and coverage metric is one such scalar value as taught in DOUGLAS.
However, DOUGLAS does not teach including a heat signature data derived from the temperature information in the feature vector.
DACOSTA teaches methods for thermal imaging a wound and determining an infection is present based on a temperature differential between different areas. (Abstract and claims 1 and 10). DACOSTA teaches that a temperature difference between different areas may indicate an infection. “As shown in the example of FIGS. 5A and 5B, an indication that the test point is warmer than the reference point may indicate the potential presence of inflammation or infection.” ([0047]). Notably, the test point and the reference point are two points within the same “target and surrounding area.” (see, e.g., [0008]-[0009]). “The example outputs are shown as color images output by a multi-modal imaging device in accordance with the present disclosure, where the output is a thermal map of the imaged target and surrounding area, with a user selected reference point and user selected test point applied to the thermal map and a temperature differential between the two points being indicated numerically as well as by a relative color scale….” ([0067]).
DACOSTA teaches that the device can analyze the data and output an indication of an infection. “The device may be further configured to analyze [image data including thermal data], correlate such data, and provide an output based on the correlation of the data, such as, for example, an indication of wound status, wound healing, wound infection, bacterial load, or other diagnostic information upon which an intervention strategy may be based.” ([0062]). Moreover, this decision-making includes considering a threshold temperature differential. “[A] temperature differential of 3° C or more is indicative of an elevated temperature which may indicate infection.” ([0063]). The temperature differential could be displayed with the thermal image. ([0078]).
Accordingly, DACOSTA teaches that heat signature data derived from the temperature information helps determine if an infection exists.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the feature vector to include heat signature data derived from the temperature information with the feature vector. RUPPERSBERG teaches that localized elevated temperatures are indicative of inflammation/infection. DOUGLAS teaches diagnostically relevant information may be combined into a feature vector. As taught by DACOSTA, temperature differentials (i.e., heat signature data) can be highly relevant in detecting signs of infection (i.e., diagnostically relevant information). Moreover, the temperature differentials are scalar values that can be derived from the same thermal image data that includes the grid-coordinate temperature values and the coverage metric. Accordingly, there would have been a reasonable expectation of success in generating feature vectors that include temperature differentials along with the grid-coordinate temperature values and a coverage metric.
With respect to claim 2, RUPPERSBERG does not explicitly teach generating an artificial intelligence model configured to detect the infections in the ear canal of the patient, and wherein applying the sensor data to the trained model comprises applying the sensor data to the artificial intelligence model.
DOUGLAS, on the other hand, teaches training a machine-learning model to detect infections and then using that trained model to detect ear infections. (see, e.g., [0176] and [0245]-[0255]).
It would have been obvious to one having ordinary skill in the art at the time of filing to generate an artificial intelligence model configured to detect the infections in the ear canal of the patient and apply the sensor data to the artificial intelligence model, as taught in DOUGLAS. One of ordinary skill in the art would have been motivated to generate (i.e., train) a machine-learning model using the tympanic-imaging device or the same type of IR images so that the trained model would be built using the same data that would subsequently be acquired to analyze a patient’s ear canals. There would have been a reasonable expectation of success as DOUGLAS teaches that AI models can be trained and used to diagnose ear infections.
With respect to claim 3, as explained above with respect to claim 1, DOUGLAS teaches applying the sensor data to the trained model comprises applying the sensor data to the artificial intelligence model as trained. “Once exam features (including features from images, clinical information or other tests) are collected and combined (either solely in a current exam, or including information from past exams), these features can be used in a machine learning system to predict the diagnosis associated with a given exam (FIG. 14A).” ([0241]). See also [0063] as explained above.
DOUGLAS also teaches training an artificial intelligence model to detect the infections in the ear canal of the patient based at least in part on historical temperature data of other patients. “Assessments of diagnosis, prognosis, etc., often benefit from general clinical information about the patient, in addition to image-based features.” ([0221]). “Generally, clinical information falls into one of two possible categories: numerical or categorical, which can be treated differently when converting them to features that are suitable for machine learning.” ([0222]). “Numerical information (e.g., age, temperature, blood pressure) can take on scalar, real-numbered values (generally greater than 0). As such, they [temperature measurements] are suitable for directly being used as features in a machine learning system….” ([0223]). “Once calculated, clinical features may be used in concert with any other kind of feature, e.g., image features.” ([0226]). Notably, image-based features would include those derived from IR images. (see, e.g., [0245]-[0255] describing a learning system that is trained using input color images.)
“The methods described herein can take the features created from all of these different data sources and assemble them into a combined feature vector (FIG. 11). This combined feature vector is suitable as input into a machine learning system for predicting, e.g., diagnosis or prognosis.” ([0230]). As is common in training machine-learning models, the temperature data may be from other patients. “Training of the system would proceed in a supervised sense, based on training with a ‘training set’ of exam features associated with known diagnosis.” ([0242]).
It would have been obvious to one having ordinary skill in the art at the time of filing to train an artificial intelligence model to detect the infections in the ear canal of the patient based at least in part on historical temperature data of other patients, as taught in DOUGLAS. One of ordinary skill in the art would have been motivated to generate (i.e., train) a machine-learning model using various features, including numerical features like temperature measurements from other patients that have been diagnosed with an ear infection, as described in DOUGLAS. There would have been a reasonable expectation of success as DOUGLAS teaches that AI models can be trained and used to diagnose ear infections.
With respect to claim 5, RUPPERSBERG does not describe sending, via a wireless communication component of the device, subsequent sensor data obtained over time to a remote system and receiving, from the remote system and via the wireless communication component, external data associated with the subsequent sensor data, and wherein detecting the signs of the infection is based at least in part on the external data.
DOUGLAS teaches continuously monitoring the ear infection (i.e., acquiring subsequent data) to determine if the intervention is effective. “A method for continuous monitoring is shown in FIG. 20. Initially, an exam is performed, a diagnosis is made (possibly with help from the automated diagnosis method, described herein) and an intervention (including, possibly, no intervention or “watchful waiting”) is prescribed (possibly with help from the automated prognosis method). After waiting for some period, e.g., one day, the exam is repeated. If the disease is improving as expected (possibly as determined with help from the automated method to determine disease stage) but has not yet resolved, the waiting period plus exam cycle may be repeated. If the disease is not improving as expected, a new intervention is prescribed based on the results of the most recent exam….” ([0266]).
DOUGLAS also teaches that the process may be carried out by a mobile device that is in communication with a remote server that at least partially analyzes the data. “The ear exam in the doctor's office (query images or videos) would be transmitted to the data analytics server, which hosts the diagnosis classification model. After processing through the diagnosis classification model, the exam analysis (with diagnosis outcome) would in turn be transmitted back to the mobile phone.” ([0258]). Embodiments may be “part of a mobile telecommunications device, which may efficiently locally process image data (e.g., for images taken with a scope (e.g., otoscope) coupled to the mobile telecommunications device such as a smartphone, and remotely communicate the information with a database, including an image database, and/or a medical server (e.g., medical records, hospital, clinic, or physician). The distribution of functions between the local (e.g., smartphone) and the remote (e.g., database) may allow more efficient processor usage, enhancing speed and reliability.” ([0446]).
It would have been obvious to one having ordinary skill in the art at the time of filing to send, via a wireless communication component of the device, subsequent sensor data obtained over time to a remote system and receive, from the remote system and via the wireless communication component, external data associated with the subsequent sensor data, and wherein detecting the signs of the infection is based at least in part on the external data, as taught in DOUGLAS. One of ordinary skill in the art would have monitored an ear infection by acquiring images of the tympanic membrane and communicating those images to a remote server for analysis, which would then communicate its results back to the user. One of ordinary skill in the art would have been motivated to perform these steps for the convenience of using a software that can reliably and quickly analyze images using a trained model, as described in DOUGLAS. There would have been a reasonable expectation of success as DOUGLAS teaches that AI models can be trained and used to diagnose ear infections.
With respect to claim 6, RUPPERSBERG does not explicitly teach storing a machine learning model trained to detect the signs of the infection from occlusion-compensated infrared data, grid-coordinate temperature values, and heat signature data; and applying the machine learning model to the occlusion-compensated thermal feature vector.
However, DOUGLAS teaches training the ML model and storing the ML model. “The methods described herein can take the features created from all of these different data sources and assemble them into a combined feature vector (FIG. 11). This combined feature vector is suitable as input into a machine learning system for predicting, e.g., diagnosis or prognosis.” ([0230]). As is common in training machine-learning models, the temperature data may be from other patients. “Training of the system would proceed in a supervised sense, based on training with a ‘training set’ of exam features associated with known diagnosis.” ([0242]). “Once exam features (including features from images, clinical information or other tests) are collected and combined (either solely in a current exam, or including information from past exams), these features can be used in a machine learning system to predict the diagnosis associated with a given exam (FIG. 14A).” ([0241]). See also [0063] as explained above. DOUGLAS also teaches storing the ML model on a device to be subsequently used for analysis. (see, e.g., [0065]).
It would have been obvious to one having ordinary skill in the art at the time of filing to train an artificial intelligence model to detect the infections in the ear canal of the patient based at least in part on temperature data of other patients, as taught in DOUGLAS, including the temperature data that forms the feature vector. One of ordinary skill in the art would have been motivated to generate (i.e., train) a machine-learning model using feature vectors, including numerical features like temperature measurements, heat signature data, and a coverage metric from other patients that have been diagnosed with an ear infection, as described in DOUGLAS. There would have been a reasonable expectation of success as DOUGLAS teaches that AI models can be trained and used to diagnose ear infections.
With respect to claim 7, RUPPERSBERG does not describe determining that the signs of the infection were detected to at least a threshold degree of confidence and causing output of an alert indicating a diagnosis of acute otitis media infection based at least in part on the signs of the infection being detected to at least the threshold degree of confidence.
DOUGLAS, on the other hand, teaches that “[o]nce exam features (including features from images, clinical information or other tests) are collected and combined (either solely in a current exam, or including information from past exams), these features can be used in a machine learning system to predict the diagnosis associated with a given exam (FIG. 14A). In this case, the machine learning system would associate a real-valued “score” associated with each possible diagnosis; the score for a given diagnosis would be related to (but not necessarily the same as) the probability of that diagnosis being accurate for the given exam. The score may also contain a confidence interval. Displaying the results of the automated diagnosis method to the user is discussed below” ([0241]).
DOUGLAS also teaches that “[f]or many diseases, there may be several ‘stages’ of the disease, ranging from the most acute form of the disease to resolution. One could also interpret the disease stage as lying on a continuum between acute and resolution, without requiring discrete stages.. If the disease stages are continuous, this method would be a form of regression and would function similarly to the automated prognosis method, except that the response for only one disease would be shown, as opposed to the response for multiple interventions as detailed herein. The response would then be the expected disease stage, which would be a real number, e.g., from 0 to 1, plus or minus a confidence interval.”
To be clear, DOUGLAS teaches determining that the signs of the infection were detected to at least a threshold degree of confidence by selecting a number for the disease stage. If the signs of the infection were different (i.e., more or less severe or otherwise indicative of a different stage), then the expected disease stage selected by DOUGLAS would have been different. In other words, DOUGLAS necessarily detects a disease stage to at least a threshold degree of confidence.
DOUGLAS also necessarily causes an output of an alert indicating a diagnosis of acute otitis media infection based at least in part on the signs of the infection when DOUGLAS outputs a response that includes the “expected disease stage, which would be a real number, e.g., from 0 to 1, plus or minus a confidence interval.” ([0264]; see also Figure 17 and [0256]: “The machine-learning model outputs a diagnosis outcome (example: normal versus AOM).”
It would have been obvious to one having ordinary skill in the art at the time of filing to s determine that the signs of the infection were detected to at least a threshold degree of confidence and cause an output of an alert indicating a diagnosis of acute otitis media infection based at least in part on the signs of the infection being detected to at least the threshold degree of confidence, as taught in DOUGLAS. One of ordinary skill in the art would have been motivated to use a threshold degree of confidence so as to not mislead the user and to output the diagnosis to the user, as described in DOUGLAS. There would have been a reasonable expectation of success as DOUGLAS teaches that AI models can be trained and used to diagnose ear infections.
With respect to claim 8 (and in light of the Section 112(b) rejection), RUPPERSBERG teaches a method for detecting ear infections (Title and Abstract). Although described in the context of determine a body core temperature, the RUPPERSBERG is also described as being capable of “diagnosing an ear disease,” which may include “reliably diagnosing e.g. an inflammation of the eardrum without the need of assistance of a skilled physician.” ([0133]). The method comprising:
receiving an indication that a portion of the device has been inserted into an ear canal of the patient. RUPPERSBERG describes a “moving mechanism” that is triggered when the device is inserted into the ear canal. “As soon as the probe cover 60 gets in contact with an inner lateral surface of the ear canal, a friction force is exerted on the probe cover 60. The friction force depends on the position of the head portion 14 within the ear canal: the friction force increases with increasing insertion depth.” ([0193]). The moving mechanism is electrically connected to one of the cameras. “In case the probe cover 60 is axially displaced, the motion detector can emit an electric signal which is transmitted to the at least one camera 40.1 or any logical unit or control unit, evoking start-up or powering of the camera 40.1. In such a way, by means of motion detection or detection of the axial position of the probe cover 60, the camera 40.1 can be powered at a time when the camera 40.1 is in visual communication with the eardrum.” ([0199]). Notably, the imaging unit and infrared sensor unit operate simultaneously. ([0165]).
receiving sensor data from the sensor array, the sensor data indicating temperature information at grid-coordinate locations of the sensor array. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]).
RUPPERSBERG does not explicitly teach generating, from the sensor data, an occlusion-compensated thermal feature vector including at least grid-coordinate temperature values, a coverage metric of the tympanic membrane by the sensor array, and heat signature data derived from the temperature information; applying the occlusion-compensated thermal feature vector to a trained model configured to detect infections in the ear canal of the patient; and detecting, based at least in part on the occlusion-compensated thermal feature vector and the trained model, signs of infection in the ear canal of the patient. Nonetheless, RUPPERSBERG:
teaches that ear infections can be identified based on temperature information. (see, e.g., [0028]: “Local inflammations also lead to a raise in temperature at the site of inflammation” and [0068]: “Inflammation of the eardrum may suggest e.g. an (bacterial/viral) infection”).
teaches that it is critical that the analysis be based on image data of the tympanic membrane and not, for example, the canal walls or obstructions (e.g. ear wax). ([0003]; see also [0009]: “In particular, it is an object of the present invention to provide an ear inspection device or a method of acquiring signals or data that allow for reliable differentiation of the eardrum.”; and [0021]: “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]).
teaches that the unwanted objects have a different temperature than the eardrum. “[T]he temperature at the surface of the eardrum (substantially corresponding to the subject's body core temperature) is higher than the temperature of the wall of the exterior ear canal and/or of earwax, hair or dirt in the exterior ear canal.” ([0021]).
teaches removing or disregarding objects that are not of interest to improve analysis. “In order to improve the identification of objects in the subject's ear, the logic unit is preferably further configured for identifying and discriminating different objects in the subject's ear, such as earwax, hair and the eardrum….” ([0031]). “[O]bjects shown in the at least on captured image may be identified (and distinguished from other objects in the subject's ear), and then the status (especially the temperature) of at least one of the identified objects is determined.” ([0133]).
DOUGLAS, on the other hand, teaches “methods and apparatuses for assisting in the acquisition and analysis of images of the tympanic membrane to provide information that may be helpful in the understanding and management of disease, such ear infection (acute otitis media).” (Abstract). DOUGLAS teaches various steps (described in greater detail below) for analyzing images of the tympanic membrane to detect signs of infection.
DOUGLAS teaches extracting features and determining the probability those features identify the tympanic membrane. ([0014]). DOUGLAS teaches generating feature vectors derived from images of tympanic membranes and inputting the feature vectors into a trained model. ([0053]-[0054]). “Machine learning methods typically work on a fixed set of features as inputs to return a response, or output. A set of features consists of scalar values that summarize the input (e.g., an image) and can be easily processed with a computer…These features together comprise a ‘feature vector’ that would be extracted from the image before being fed as input into a machine learning method to predict some response (e.g., diagnosis, prognosis, etc.).” ([0210]; see also [0229]-[0230] describing how features are combined into a feature vector).
DOUGLAS further teaches that the features must be carefully selected and extracted. (see, e.g., [0211]). “[I]mage features are typically extracted via a feature extraction method that is resistant to translation, scaling and other deformations and is designed specifically with the image subject in mind.” ([0212]). DOUGLAS then describes various features that could be used with images of tympanic membranes. DOUGLAS even describes temperature as one example of “numerical information” that can be represented by scalar, real-numbered values that are “suitable for directly being used as features in a machine learning system….”
Similar to RUPPERSBERG, DOUGLAS is concerned with a sufficient amount of the tympanic membrane being visible and, as such, teaches guiding the user so that the tympanic membrane is sufficiently viewable. “In general, a method of guidance or an apparatus for guiding a subject to take an image may examine images (digital images) of a patient's ear canal being taken by the user, e.g., operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%,….” ([0036]). DOUGLAS is also concerned with obstructions. “Obstructions within the ear canal, including cerumen or foreign bodies may either prevent the ear exam from being completed successfully (because they partially or completely occlude the tympanic membrane) or may present a hazard during the otoscopic examination because they may be pushed by the speculum deeper into the ear.” ([0328]).
DOUGLAS teaches that portions of the image that do not include the eardrum should be excluded. First, an image is analyzed to identify those subregions that are likely part of the tympanic membrane. (see, e.g., [0014]-[0015]). When generating the feature vector, the portions that are not part of the tympanic membrane can be excluded. “The image is divided into blocks of a certain size (example 32×32) and regions that have certain characteristics that do not imply the presence of eardrum could be excluded automatically.” ([0248]). DOUGLAS further teaches that obstructions could be identified in a similar manner as to how regions of the tympanic membrane are identified. ([0328]-[0329]).
Accordingly, DOUGLAS teaches generating, from the sensor data, an occlusion-compensated thermal feature vector. DOUGLAS also teaches applying the occlusion-compensated thermal feature vector to a trained model configured to detect infections in the ear canal of the patient (see, e.g., [0256], see also [0252]-[0254]) and detecting, based at least in part on the occlusion-compensated thermal feature vector and the trained model, signs of infection in the ear canal of the patient. (see, e.g., [0256], see also [0241]-[0242]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG computing device to generate a feature vector that includes sensor data (i.e., grid-coordinate temperature values) and that was compensated for to remove unnecessary information. One would be motivated to use a feature vector as an input to machine-learning model as ML models are capable of detecting signs of ear infection based on image data. Moreover, both RUPPERSBERG and DOUGLAS teach identifying and excluding unwanted image data. Prior to generating the feature vector the grid-coordinates associated with canal walls and earwax or other objects would be excluded as taught by DOUGLAS. There would have been a reasonable expectation of success as DOUGLAS teaches excluding regions that do not correspond to the tympanic membrane.
Moreover, it would have been obvious to one having ordinary skill in the art at the time of filing to modify the feature vector to include a coverage metric. DOUGLAS teaches that one diagnostically relevant piece of information is the amount of the tympanic membrane that is captured within the image. After excluding unwanted data associated with the canal wall and obstructions, one having ordinary skill in the art would include, with the feature vector, a coverage metric representing how much of the tympanic membrane is represented by the feature vector. For example, if the image field of view and obstructions only reveal 50% of the tympanic membrane, then the feature vector will include a coverage metric to indicate that the grid-coordinate temperature values do not represent the entire tympanic membrane. As the percentage of the tympanic membrane decreases, the confidence score in the output of the ML model would necessarily decrease. There would have been a reasonable expectation of success as DOUGLAS teaches that feature vectors are a combination of relevant scalar values and coverage metric is one such scalar value as taught in DOUGLAS.
However, DOUGLAS does not teach including a heat signature data derived from the temperature information in the feature vector.
DACOSTA teaches methods for thermal imaging a wound and determining an infection is present based on a temperature differential between different areas. (Abstract and claims 1 and 10). DACOSTA teaches that a temperature difference between different areas may indicate an infection. “As shown in the example of FIGS. 5A and 5B, an indication that the test point is warmer than the reference point may indicate the potential presence of inflammation or infection.” ([0047]). Notably, the test point and the reference point are two points within the same “target and surrounding area.” (see, e.g., [0008]-[0009]). “The example outputs are shown as color images output by a multi-modal imaging device in accordance with the present disclosure, where the output is a thermal map of the imaged target and surrounding area, with a user selected reference point and user selected test point applied to the thermal map and a temperature differential between the two points being indicated numerically as well as by a relative color scale….” ([0067]).
DACOSTA teaches that the device can analyze the data and output an indication of an infection. “The device may be further configured to analyze [image data including thermal data], correlate such data, and provide an output based on the correlation of the data, such as, for example, an indication of wound status, wound healing, wound infection, bacterial load, or other diagnostic information upon which an intervention strategy may be based.” ([0062]). Moreover, this decision-making includes considering a threshold temperature differential. “[A] temperature differential of 3° C or more is indicative of an elevated temperature which may indicate infection.” ([0063]). The temperature differential could be displayed with the thermal image. ([0078]).
Accordingly, DACOSTA teaches that heat signature data derived from the temperature information helps determine if an infection exists.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the feature vector to include heat signature data derived from the temperature information with the feature vector. RUPPERSBERG teaches that localized elevated temperatures are indicative of inflammation/infection. DOUGLAS teaches diagnostically relevant information may be combined into a feature vector. As taught by DACOSTA, temperature differentials (i.e., heat signature data) can be highly relevant in detecting signs of infection (i.e., diagnostically relevant information). Moreover, the temperature differentials are scalar values that can be derived from the same thermal image data that includes the grid-coordinate temperature values and the coverage metric. Accordingly, there would have been a reasonable expectation of success in generating feature vectors that include temperature differentials along with the grid-coordinate temperature values and a coverage metric.
With respect to claim 9, RUPPERSBERG teaches simultaneously capturing data from multiple infrared sensing points from the sensor array to create a thermal profile of a tympanic membrane of the patient. The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). RUPPERSBERG also describes technology enabling thermal images of sufficient resolution. “Therefore, wafer-level imaging technology allows obtaining images (of the temperature distribution and/or of light in the visual range) of “sufficient” resolution of the eardrum, e.g. images of 250 pixels×250 pixels, with a footprint of the camera (including a lens) of only about 1 mm×1 mm or even smaller.” ([0052]).
With respect to claim 12, RUPPERSBERG does not teach determining a first temperature reading of a first portion of the ear canal, determining a second temperature reading of a second portion of the ear canal, determining a temperature differential of the first temperature reading and the second temperature reading, and wherein detecting the signs of the infection is based at least in part on the temperature differential satisfying a pre-set temperature differential criteria.
As discussed above, however, DACOSTA teaches that a temperature difference between different areas may indicate an infection. “As shown in the example of FIGS. 5A and 5B, an indication that the test point is warmer than the reference point may indicate the potential presence of inflammation or infection.” ([0047]). Notably, the test point and the reference point are two points within the same “target and surrounding area.” (see, e.g., [0008]-[0009]). “The example outputs are shown as color images output by a multi-modal imaging device in accordance with the present disclosure, where the output is a thermal map of the imaged target and surrounding area, with a user selected reference point and user selected test point applied to the thermal map and a temperature differential between the two points being indicated numerically as well as by a relative color scale….” ([0067]).
DACOSTA teaches that the device can analyze the data and output an indication of an infection. “The device may be further configured to analyze [image data including thermal data], correlate such data, and provide an output based on the correlation of the data, such as, for example, an indication of wound status, wound healing, wound infection, bacterial load, or other diagnostic information upon which an intervention strategy may be based.” ([0062]). Moreover, this decision-making includes considering a threshold temperature differential. “[A] temperature differential of 3° C or more is indicative of an elevated temperature which may indicate infection.” ([0063]). The temperature differential could be displayed with the thermal image. ([0078]).
It would have been obvious to one having ordinary skill in the art at the time of filing to determine a temperature differential of a first temperature reading and a second temperature reading from different portions of the ear canal and detect the signs of the infection is based at least in part on the temperature differential satisfying a pre-set temperature differential criteria. RUPPERSBERG teaches that localized elevated temperatures are indicative of inflammation/infection. As taught by DACOSTA, temperature differentials can be highly relevant in detecting signs of infection and they can be compared to pre-set temperature differentials to determine if an infection exists. There would have been a reasonable expectation of success as DACOSTA teaches that temperature differentials can be computed based on the thermal images.
With respect to claim 15 (and in light of the Section 112(b) rejection), RUPPERSBERG teaches a system for detecting ear infections (Title and Abstract). Although described in the context of determine a body core temperature, the RUPPERSBERG is also described as being capable of “diagnosing an ear disease,” which may include “reliably diagnosing e.g. an inflammation of the eardrum without the need of assistance of a skilled physician.” ([0133]). The system comprising:
one or more processors. Claim 3 recites “a logic unit configured for receiving and processing signals from the infrared sensor unit….”.
a sensor array configured to measure tympanic temperatures across multiple points on a tympanic membrane of a patient. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). RUPPERSBERG also describes technology enabling thermal images of sufficient resolution. “Therefore, wafer-level imaging technology allows obtaining images (of the temperature distribution and/or of light in the visual range) of “sufficient” resolution of the eardrum, e.g. images of 250 pixels×250 pixels, with a footprint of the camera (including a lens) of only about 1 mm×1 mm or even smaller.” ([0052]).
a housing configured to contain the one or more processors and the sensor array, the housing having a distal end configured for at least partial insertion into an ear canal of the patient. Figure 1 of RUPPERSBERG shows an ear inspection device 10 having a head portion 14 that includes a logic unit 44. The head portion 14 includes a distal end 18 that “is adapted to be introduced into a subject's ear canal.” ([0155]).
receiving an indication that a portion of the device has been inserted into an ear canal of the patient. RUPPERSBERG describes a “moving mechanism” that is triggered when the device is inserted into the ear canal. “As soon as the probe cover 60 gets in contact with an inner lateral surface of the ear canal, a friction force is exerted on the probe cover 60. The friction force depends on the position of the head portion 14 within the ear canal: the friction force increases with increasing insertion depth.” ([0193]). The moving mechanism is electrically connected to one of the cameras. “In case the probe cover 60 is axially displaced, the motion detector can emit an electric signal which is transmitted to the at least one camera 40.1 or any logical unit or control unit, evoking start-up or powering of the camera 40.1. In such a way, by means of motion detection or detection of the axial position of the probe cover 60, the camera 40.1 can be powered at a time when the camera 40.1 is in visual communication with the eardrum.” ([0199]). Notably, the imaging unit and infrared sensor unit operate simultaneously. ([0165]).
receiving sensor data from the sensor array, the sensor data indicating temperature information at grid-coordinate locations of the sensor array. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]).
RUPPERSBERG does not explicitly teach that the device includes a non-transitory computer-readable media, housed in the housing, storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. Nonetheless, RUPPERSBERG a logic unit 44 that processes infrared/imaging data and is configured to evaluate the images. (see, e.g., claims 3 and 4). The logic unit is stored in a housing. (see Figure 1).
DOUGLAS, on the other hand, teaches “methods and apparatuses for assisting in the acquisition and analysis of images of the tympanic membrane to provide information that may be helpful in the understanding and management of disease, such ear infection (acute otitis media).” (Abstract). DOUGLAS teaches various steps (described in greater detail below) for analyzing images of the tympanic membrane to detect signs of infection. DOUGLAS further teaches that these steps may be carried out by a processor that executes instructions stored on a computer-readable medium. “For example, an apparatus may also include an otoscope or otoscope adapter (lens portion) to connected to a mobile telecommunciations device and a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor, that when executed by the processor causes the processor to….” ([0031]; see also [0032] and [0051]).
It would have been obvious to one having ordinary skill in the art at the time of filing to add a non-transitory computer-readable media to the RUPPERSBERG device, housed in the housing and storing instructions, as recited. One of ordinary skill in the art could have added a non-transitory computer-readable medium within the housing using known methods. One of ordinary skill in the art would have recognized that the results of adding the computer-readable medium would be predictable.
RUPPERSBERG does not explicitly teach generating, from the sensor data, an occlusion-compensated thermal feature vector including at least grid-coordinate temperature values, a coverage metric of the tympanic membrane by the sensor array, and heat signature data derived from the temperature information; applying the occlusion-compensated thermal feature vector to a trained model configured to detect infections in the ear canal of the patient; and detecting, based at least in part on the occlusion-compensated thermal feature vector and the trained model, signs of infection in the ear canal of the patient. Nonetheless, RUPPERSBERG:
teaches that ear infections can be identified based on temperature information. (see, e.g., [0028]: “Local inflammations also lead to a raise in temperature at the site of inflammation” and [0068]: “Inflammation of the eardrum may suggest e.g. an (bacterial/viral) infection”).
teaches that it is critical that the analysis be based on image data of the tympanic membrane and not, for example, the canal walls or obstructions (e.g. ear wax). ([0003]; see also [0009]: “In particular, it is an object of the present invention to provide an ear inspection device or a method of acquiring signals or data that allow for reliable differentiation of the eardrum.”; and [0021]: “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]).
teaches that the unwanted objects have a different temperature than the eardrum. “[T]he temperature at the surface of the eardrum (substantially corresponding to the subject's body core temperature) is higher than the temperature of the wall of the exterior ear canal and/or of earwax, hair or dirt in the exterior ear canal.” ([0021]).
teaches removing or disregarding objects that are not of interest to improve analysis. “In order to improve the identification of objects in the subject's ear, the logic unit is preferably further configured for identifying and discriminating different objects in the subject's ear, such as earwax, hair and the eardrum….” ([0031]). “[O]bjects shown in the at least on captured image may be identified (and distinguished from other objects in the subject's ear), and then the status (especially the temperature) of at least one of the identified objects is determined.” ([0133]).
DOUGLAS teaches extracting features and determining the probability those features identify the tympanic membrane. ([0014]). DOUGLAS teaches generating feature vectors derived from images of tympanic membranes and inputting the feature vectors into a trained model. ([0053]-[0054]). “Machine learning methods typically work on a fixed set of features as inputs to return a response, or output. A set of features consists of scalar values that summarize the input (e.g., an image) and can be easily processed with a computer…These features together comprise a ‘feature vector’ that would be extracted from the image before being fed as input into a machine learning method to predict some response (e.g., diagnosis, prognosis, etc.).” ([0210]; see also [0229]-[0230] describing how features are combined into a feature vector).
DOUGLAS further teaches that the features must be carefully selected and extracted. (see, e.g., [0211]). “[I]mage features are typically extracted via a feature extraction method that is resistant to translation, scaling and other deformations and is designed specifically with the image subject in mind.” ([0212]). DOUGLAS then describes various features that could be used with images of tympanic membranes. DOUGLAS even describes temperature as one example of “numerical information” that can be represented by scalar, real-numbered values that are “suitable for directly being used as features in a machine learning system….”
Similar to RUPPERSBERG, DOUGLAS is concerned with a sufficient amount of the tympanic membrane being visible and, as such, teaches guiding the user so that the tympanic membrane is sufficiently viewable. “In general, a method of guidance or an apparatus for guiding a subject to take an image may examine images (digital images) of a patient's ear canal being taken by the user, e.g., operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%,….” ([0036]). DOUGLAS is also concerned with obstructions. “Obstructions within the ear canal, including cerumen or foreign bodies may either prevent the ear exam from being completed successfully (because they partially or completely occlude the tympanic membrane) or may present a hazard during the otoscopic examination because they may be pushed by the speculum deeper into the ear.” ([0328]).
DOUGLAS teaches that portions of the image that do not include the eardrum should be excluded. First, an image is analyzed to identify those subregions that are likely part of the tympanic membrane. (see, e.g., [0014]-[0015]). When generating the feature vector, the portions that are not part of the tympanic membrane can be excluded. “The image is divided into blocks of a certain size (example 32×32) and regions that have certain characteristics that do not imply the presence of eardrum could be excluded automatically.” ([0248]). DOUGLAS further teaches that obstructions could be identified in a similar manner as to how regions of the tympanic membrane are identified. ([0328]-[0329]).
Accordingly, DOUGLAS teaches generating, from the sensor data, an occlusion-compensated thermal feature vector. DOUGLAS also teaches applying the occlusion-compensated thermal feature vector to a trained model configured to detect infections in the ear canal of the patient (see, e.g., [0256], see also [0252]-[0254]) and detecting, based at least in part on the occlusion-compensated thermal feature vector and the trained model, signs of infection in the ear canal of the patient. (see, e.g., [0256], see also [0241]-[0242]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG computing device to generate a feature vector that includes sensor data (i.e., grid-coordinate temperature values) and that was compensated for to remove unnecessary information. One would be motivated to use a feature vector as an input to machine-learning model as ML models are capable of detecting signs of ear infection based on image data. Moreover, both RUPPERSBERG and DOUGLAS teach identifying and excluding unwanted image data. Prior to generating the feature vector the grid-coordinates associated with canal walls and earwax or other objects would be excluded as taught by DOUGLAS. There would have been a reasonable expectation of success as DOUGLAS teaches excluding regions that do not correspond to the tympanic membrane.
Moreover, it would have been obvious to one having ordinary skill in the art at the time of filing to modify the feature vector to include a coverage metric. DOUGLAS teaches that one diagnostically relevant piece of information is the amount of the tympanic membrane that is captured within the image. After excluding unwanted data associated with the canal wall and obstructions, one having ordinary skill in the art would include, with the feature vector, a coverage metric representing how much of the tympanic membrane is represented by the feature vector. For example, if the image field of view and obstructions only reveal 50% of the tympanic membrane, then the feature vector will include a coverage metric to indicate that the grid-coordinate temperature values do not represent the entire tympanic membrane. As the percentage of the tympanic membrane decreases, the confidence score in the output of the ML model would necessarily decrease. There would have been a reasonable expectation of success as DOUGLAS teaches that feature vectors are a combination of relevant scalar values and coverage metric is one such scalar value as taught in DOUGLAS.
However, DOUGLAS does not teach including a heat signature data derived from the temperature information in the feature vector.
DACOSTA teaches methods for thermal imaging a wound and determining an infection is present based on a temperature differential between different areas. (Abstract and claims 1 and 10). DACOSTA teaches that a temperature difference between different areas may indicate an infection. “As shown in the example of FIGS. 5A and 5B, an indication that the test point is warmer than the reference point may indicate the potential presence of inflammation or infection.” ([0047]). Notably, the test point and the reference point are two points within the same “target and surrounding area.” (see, e.g., [0008]-[0009]). “The example outputs are shown as color images output by a multi-modal imaging device in accordance with the present disclosure, where the output is a thermal map of the imaged target and surrounding area, with a user selected reference point and user selected test point applied to the thermal map and a temperature differential between the two points being indicated numerically as well as by a relative color scale….” ([0067]).
DACOSTA teaches that the device can analyze the data and output an indication of an infection. “The device may be further configured to analyze [image data including thermal data], correlate such data, and provide an output based on the correlation of the data, such as, for example, an indication of wound status, wound healing, wound infection, bacterial load, or other diagnostic information upon which an intervention strategy may be based.” ([0062]). Moreover, this decision-making includes considering a threshold temperature differential. “[A] temperature differential of 3° C or more is indicative of an elevated temperature which may indicate infection.” ([0063]). The temperature differential could be displayed with the thermal image. ([0078]).
Accordingly, DACOSTA teaches that heat signature data derived from the temperature information helps determine if an infection exists.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the feature vector to include heat signature data derived from the temperature information with the feature vector. RUPPERSBERG teaches that localized elevated temperatures are indicative of inflammation/infection. DOUGLAS teaches diagnostically relevant information may be combined into a feature vector. As taught by DACOSTA, temperature differentials (i.e., heat signature data) can be highly relevant in detecting signs of infection (i.e., diagnostically relevant information). Moreover, the temperature differentials are scalar values that can be derived from the same thermal image data that includes the grid-coordinate temperature values and the coverage metric. Accordingly, there would have been a reasonable expectation of success in generating feature vectors that include temperature differentials along with the grid-coordinate temperature values and a coverage metric.
With respect to claim 16, RUPPERSBERG teaches simultaneously capturing data from multiple infrared sensing points from the sensor array to create a thermal profile of a tympanic membrane of the patient. The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). RUPPERSBERG also describes technology enabling thermal images of sufficient resolution. “Therefore, wafer-level imaging technology allows obtaining images (of the temperature distribution and/or of light in the visual range) of “sufficient” resolution of the eardrum, e.g. images of 250 pixels×250 pixels, with a footprint of the camera (including a lens) of only about 1 mm×1 mm or even smaller.” ([0052]).
With respect to claim 19, RUPPERSBERG does not teach determining a first temperature reading of a first portion of the ear canal, determining a second temperature reading of a second portion of the ear canal, determining a temperature differential of the first temperature reading and the second temperature reading, and wherein detecting the signs of the infection is based at least in part on the temperature differential satisfying a pre-set temperature differential criteria.
As discussed above with respect to claim 12, however, DACOSTA teaches that a temperature difference between different areas may indicate an infection. “As shown in the example of FIGS. 5A and 5B, an indication that the test point is warmer than the reference point may indicate the potential presence of inflammation or infection.” ([0047]). Notably, the test point and the reference point are two points within the same “target and surrounding area.” (see, e.g., [0008]-[0009]). “The example outputs are shown as color images output by a multi-modal imaging device in accordance with the present disclosure, where the output is a thermal map of the imaged target and surrounding area, with a user selected reference point and user selected test point applied to the thermal map and a temperature differential between the two points being indicated numerically as well as by a relative color scale….” ([0067]).
DACOSTA teaches that the device can analyze the data and output an indication of an infection. “The device may be further configured to analyze [image data including thermal data], correlate such data, and provide an output based on the correlation of the data, such as, for example, an indication of wound status, wound healing, wound infection, bacterial load, or other diagnostic information upon which an intervention strategy may be based.” ([0062]). Moreover, this decision-making includes considering a threshold temperature differential. “[A] temperature differential of 3° C or more is indicative of an elevated temperature which may indicate infection.” ([0063]). The temperature differential could be displayed with the thermal image. ([0078]).
It would have been obvious to one having ordinary skill in the art at the time of filing to determine a temperature differential of a first temperature reading and a second temperature reading from different portions of the ear canal and detect the signs of the infection is based at least in part on the temperature differential satisfying a pre-set temperature differential criteria. RUPPERSBERG teaches that localized elevated temperatures are indicative of inflammation/infection. As taught by DACOSTA, temperature differentials can be highly relevant in detecting signs of infection and they can be compared to pre-set temperature differentials to determine if an infection exists. There would have been a reasonable expectation of success as DACOSTA teaches that temperature differentials can be computed based on the thermal images.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2015/0351637 A1 (hereinafter “RUPPERSBERG”) in view of U.S. Patent Appl. Publ. No. 2015/0065803 A1 (hereinafter “DOUGLAS”) and U.S. Patent Appl. Publ. No. 2024/0366145 A1 (hereinafter “DACOSTA”) as applied to claim 1 above, and further in view of U.S. Patent Appl. Publ. No. 2014/0140368 A1 (hereinafter “YILDIZYAN”).
With respect to claim 4, RUPPERSBERG does not explicitly teach storing correction parameters including at least one of ambient-temperature compensation parameters, occlusion-compensation parameters, or tympanic-versus-axillary comparison parameters; and applying the correction parameters to generate the occlusion-compensated thermal feature vector.
In the same field of endeavor, YILDIZYAN teaches a non-contact medical thermometer is disclosed that includes an IR sensor assembly and has distance sensing and compensation. (Title and Abstract). YILDIZYAN teaches detecting a temperature value of the target object, which can be an eardrum, an ambient temperature, and a distance between the thermometer and target. “Microprocessor 16 is configured to, among other things, determine a temperature value of the target based on the signal from IR sensor package/assembly 12, determine an ambient environment or thermometer temperature, and to determine a distance value corresponding to the distance between IR thermometer 10 and the target using a correlation routine based on the signal from distance sensor 14 and the characteristics of the reflected radiation.” ([0016]). In addition to the above, the memory can include compensation information. “This predetermined compensation information may be empirically predetermined by performing clinical tests. These clinical tests may relate the detected temperature of a target (e.g., forehead), the distance of the IR thermometer from the target, as well as the actual temperature of the target and the ambient environment or thermometer temperature.” ([0017]). The compensation information enables using the thermometer in environments having various ambient temperatures. ([0017]). The compensation information can be from a lookup table. ([0022]-[0023]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to store and apply correction parameters, including ambient-temperature parameters, to the feature vector. One of ordinary skill in the art would have been motivated to use correction parameters for ambient temperatures so that the device could be used in environments with different temperatures. There would have been a reasonable expectation of success as YILDIZYAN teaches that compensation parameters can be stored to compensate for different ambient temperatures.
Claims 10, 11, 13, 17, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2015/0351637 A1 (hereinafter “RUPPERSBERG”) in view of U.S. Patent Appl. Publ. No. 2015/0065803 A1 (hereinafter “DOUGLAS”) and U.S. Patent Appl. Publ. No. 2024/0366145 A1 (hereinafter “DACOSTA”) as applied to claims 8 and 15 above, and further in view of Doğan, Hatice Hilal, et al. “Comparison of axillary and tympanic temperature measurements in children diagnosed with acute otitis media.” International Journal of Pediatrics 2016 (hereinafter “DOGAN”) (previously cited).
With respect to claims 10 and 17, RUPPERSBERG does not teach generating a comparison of a sensitivity and a specificity of tympanic temperature readings from the sensor data to axillary temperature readings, and wherein detecting the signs of the infection is based at least in part on the comparison.
With respect to claims 10 and 17, DOGAN was a study conducted to determine if acute otitis media (AOM) affects the accuracy of tympanic temperature measurements and to also compare the tympanic temperature measurements to measurements from axillary thermometers. (Abstract). The study included patients with single-side AOM (i.e., only one ear was infected). (p.2, first full paragraph of left column). “Tympanic measurements were taken from both ears….” (p.2, middle of left column). DOGAN concluded that, “[i]n patients with AOM, infected ears had higher temperatures than normal ears with a mean of 0.48±0.01 °C.” (Abstract). “We suggest that the higher tympanic temperatures, approximately 0.5 °C in our study, in infected ears may aid in diagnosis of patients with fever without a source in pediatric clinics.” (Abstract). Moreover, “[c]ompared with axillary temperature, the sensitivity of tympanic temperature in the infected ear was 91.7% and the specificity was 74.8%.” (Abstract). DOGAN stated that “[c]omparisons of axillary and tympanic temperatures in children with AOM during the active infection concluded higher tympanic temperatures in infected ears.” (Abstract).
It would have been obvious to one having ordinary skill in the art at the time of filing to acquire tympanic temperature readings and axillary temperature readings from patients and to compare the sensitivity and the specificity of the readings, as taught in DOGAN. One would be motivated to use the same device and methodology to generate this data for training the diagnostic model, as taught in DOUGLAS, such that the detection of the signs of the infection is based at least in part on the comparison. For example, tympanic temperature readings that are above a threshold number while the axillary temperature readings are below a threshold number could confirm an ear infection. There would have been a reasonable expectation of success as DOGAN teaches that one can acquire tympanic and axillary measurements for comparison. Moreover, DOGAN specifically teaches that the comparison provides a sensitivity of at least 91.7% and a specificity of at least 74.8% with respect to fever detection.
With respect to claims 11 and 18 (depending from claims 10 and 17, respectively), RUPPERSBERG does not teach generating a validation output including sensitivity and specificity values determined from a validation dataset associated with tympanic temperature readings and axillary temperature readings, and wherein the trained model is updated based at least in part on the validation output.
However, DOUGLAS teaches generating a validation output including sensitivity and specificity values determined from a validation dataset associated with tympanic images. “The classifiers would be trained and tested using k-fold cross-validation or a leave one out cross-validation or some other method. The final trained model is saved for the prediction stage.” ([0254]). “Training of the system would proceed in a supervised sense, based on training with a “training set” of exam features associated with known diagnosis. Confirmation of low prediction error would be confirmed by testing the trained system in an independent benchmark test set that was not otherwise involved in the training.” ([0242]). DOUGLAS specifically teaches that sensitivity and specificity are validation metrics. “The efficacy of the cross-validated model based on the diagnosis outcome could be evaluated using a few performance metrics. Metrics may include the Area under the curve (AUC) of the Receiver Operating Characteristics (ROC) Curve, accuracy, sensitivity, specificity, positive predictive value, negative predictive value etc.” ([0255]). DOUGLAS also teaches that the training can involve evaluating candidate model/features through validation and producing a final model. (see, e.g., [0252]-[0253]).
It would have been obvious to one having ordinary skill in the art at the time of filing to use DOGAN’s tympanic/axillary AOM data as validation data in DOUGLAS’s validation methodology. One would be motivated to make this modification because DOGAN teaches that the relationship between tympanic and axillary temperatures provide diagnostically relevant information and DOUGLAS teaches that diagnostic models for acute otitis media can be evaluated using validation data sets and sensitivity/specificity metrics. There would have been a reasonable expectation of success as DOUGLAS teaches that diagnostic models can be evaluated with validation sets.
With respect to claims 13 and 20, RUPPERSBERG does not teach storing reference temperature values associated with the infection and comparing the reference temperature values to the sensor data, wherein detecting the signs of the infection is based at least in part on comparing the reference temperature values to the sensor data.
In single-sided acute otitis media, DOGAN acquired temperature readings from both ears. DOGAN concluded that, “[i]n patients with AOM, infected ears had higher temperatures than normal ears with a mean of 0.48±0.01 °C.” (Abstract). “We suggest that the higher tympanic temperatures, approximately 0.5 °C in our study, in infected ears may aid in diagnosis of patients with fever without a source in pediatric clinics.” (Abstract). Accordingly, comparing the infected ear’s temperature to the temperature of the non-infected ear (i.e., the stored reference temperature) could aid in diagnosis.
It would have been obvious to one having ordinary skill in the art at the time of filing to compare a stored reference temperature value (i.e., the tympanic temperature of the non-infected ear) to the sensor data (i.e., the tympanic temperature of the infected ear). One would have been motivated to make this comparison because detecting a difference in about 0.5 °C is indicative of an ear infection. There would have been a reasonable expectation of success as DOGAN teaches that this comparison can be performed and can help in diagnosing acute otitis media.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2015/0351637 A1 (hereinafter “RUPPERSBERG”) in view of U.S. Patent Appl. Publ. No. 2015/0065803 A1 (hereinafter “DOUGLAS”) and U.S. Patent Appl. Publ. No. 2024/0366145 A1 (hereinafter “DACOSTA”) as applied to claim 1 above, and further in view of U.S. Patent Appl. Publ. No. 2013/0235901 A1 (hereinafter “SHIN”).
With respect to claim 14, RUPPERSBERG teaches receiving multiple temperature readings across a tympanic membrane surface of the patient. ([0050]-[0052]). However, HARR does not explicitly teach averaging the multiple temperature readings such that measurement errors are minimized, and wherein detecting the signs of the infection is based at least in part on the averaged value.
In the same field of endeavor, SHIN teaches a portable device having an infrared array sensor that acquires a thermal distribution of a person. (Abstract and [0019]). SHIN notes that detecting thermal distributions can be difficult because it is necessary to have the correct positioning of the device. ([0020]). “If a distance and angle for measurement is unsuitably adjusted, it will result in a serious error in detecting temperature and/or thermal distribution from a subject.” ([0020]). To address the issue, SHIN first determines if pixels are “valid” and then determines an average of the valid pixels. Pixels can be valid if they are within a target region or within a temperature range. ([0078], [0080, [0087]). A “resultant temperature value” is calculated by averaging the valid pixels. (see, e.g., [0107]). The “resultant temperature value” represents the temperature of the subject. ([0107]). Notably, to minimize error, SHIN teaches excluding certain temperature values. “[I]n the course of calculating the resultant temperature value from the average temperature value, controller 45 may exclude the highest and lowest temperature values to stabilize the average, or grant a higher weight (e.g. 120%) to temperature values of pixels disposed at the center of the valid pixels while a lower weight (e.g. 80%) to temperature values of peripheral pixels.” ([0108]; see also [0098]: “for data accuracy”).
It would have been obvious to average the multiple temperature readings so that measurement errors are minimized when obtaining an average value of the subject. The average value can then be used as a temperature to determine if the patient has an infection as taught in DOUGLAS. ([0221]). One would have been motivated to use the averaged value so that a more precise measurement could be used when detecting signs of infection. There would have been a reasonable expectation of success as SHIN teaches that multiple temperature readings can be averaged.
RESPONSE TO APPLICANT’S ARGUMENTS
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new grounds of rejection do not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Prior Art of Record
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Kashani et al. “Shortwave infrared otoscopy for diagnosis of middle ear effusions: A machine-learning-based approach.” Scientific Reports 11.1 (2021): 12509. KASHANI describes using AI to detect middle ear effusions using shortwave infrared otoscopy. The ML model was able to use SWIR otoscopy images in detecting effusion “offering accuracy of 90.3%, sensitivity of 90.5% and specificity of 90.1%.” (Discussion).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON P GROSS whose telephone number is (571)272-1386. The examiner can normally be reached Monday-Friday 9:00-5:00CT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne M. Kozak can be reached at (571) 270-5284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON P GROSS/Examiner, Art Unit 3797
/SERKAN AKAR/Primary Examiner, Art Unit 3797
Prosecution Insights