DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims and Rejections
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Claims 1, 8, 14, 16, and 17 have been amended.
Claims 1-20 are currently pending.
In light of the claim amendments, the Section 101 rejection has been withdrawn.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5, 6, 8-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2015/0351637 A1 (hereinafter “RUPPERSBERG”) and U.S. Patent Appl. Publ. No. 2013/0296685 A1 (hereinafter “TSUBOI”) and U.S. Patent Appl. Publ. No. 2015/0065803 A1 (hereinafter “DOUGLAS”).
With respect to claim 1, RUPPERSBERG teaches a device for detecting ear infections (Title and Abstract). Although described in the context of determine a body core temperature, the RUPPERSBERG is also described as being capable of “diagnosing an ear disease,” which may include “reliably diagnosing e.g. an inflammation of the eardrum without the need of assistance of a skilled physician.” ([0133]). The device comprising:
one or more processors. Claim 3 recites “a logic unit configured for receiving and processing signals from the infrared sensor unit….”.
a sensor array configured to measure tympanic temperatures across multiple points on a tympanic membrane of a patient. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). RUPPERSBERG also describes technology enabling thermal images of sufficient resolution. “Therefore, wafer-level imaging technology allows obtaining images (of the temperature distribution and/or of light in the visual range) of “sufficient” resolution of the eardrum, e.g. images of 250 pixels×250 pixels, with a footprint of the camera (including a lens) of only about 1 mm×1 mm or even smaller.” ([0052]).
a housing configured to contain the one or more processors and the sensor array, the housing having a distal end configured for at least partial insertion into an ear canal of the patient. Figure 1 of RUPPERSBERG shows an ear inspection device 10 having a head portion 14 that includes a logic unit 44. The head portion 14 includes a distal end 18 that “is adapted to be introduced into a subject's ear canal.” ([0155]).
receiving an indication that a portion of the device has been inserted into an ear canal of the patient. RUPPERSBERG describes a “moving mechanism” that is triggered when the device is inserted into the ear canal. “As soon as the probe cover 60 gets in contact with an inner lateral surface of the ear canal, a friction force is exerted on the probe cover 60. The friction force depends on the position of the head portion 14 within the ear canal: the friction force increases with increasing insertion depth.” ([0193]). The moving mechanism is electrically connected to one of the cameras. “In case the probe cover 60 is axially displaced, the motion detector can emit an electric signal which is transmitted to the at least one camera 40.1 or any logical unit or control unit, evoking start-up or powering of the camera 40.1. In such a way, by means of motion detection or detection of the axial position of the probe cover 60, the camera 40.1 can be powered at a time when the camera 40.1 is in visual communication with the eardrum.” ([0199]). Notably, the imaging unit and infrared sensor unit operate simultaneously. ([0165]).
receiving sensor data from the sensor array, the sensor data indicating temperature information at grid-coordinate locations of the sensor array. The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]).
RUPPERSBERG does not explicitly teach that the device includes a non-transitory computer-readable media, housed in the housing, storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. Nonetheless, RUPPERSBERG a logic unit 44 that processes infrared/imaging data and is configured to evaluate the images. (see, e.g., claims 3 and 4).
In the same field of endeavor, TSUBOI teaches “a type of clinical thermometer that measures a body temperature by measuring radiant heat emitted from an eardrum….” ([0002]). TSUBOI teaches that it is challenging to acquire measurements of the eardrum continuously or over a longer period of time due to the thermometer being incorrectly positioned. ([0004]-[0005]). To address these challenges, TSUBOI teaches “an eardrum recognition unit configured to recognize a position of an eardrum based on image information regarding the eardrum, a temperature measurement unit configured to acquire a temperature within an external ear canal including the eardrum, and a temperature processing unit configured to determine a temperature of the eardrum based on a recognition result of the eardrum recognition unit and a measured temperature of the temperature measurement unit.” ([0007]). TSUBOI teaches a computer program that can cause a computer to perform the operations of these processing units. ([0009], [0131], [0132]).
It would have been obvious to one having ordinary skill in the art at the time of filing to add a non-transitory computer-readable media to the RUPPERSBERG device, housed in the housing and storing instructions, as recited. One of ordinary skill in the art could have added a non-transitory computer-readable medium within the housing using known methods. One of ordinary skill in the art would have recognized that the results of adding the computer-readable medium would be predictable.
RUPPERSBERG does not explicitly teach that the device is configured to determine, from the temperature information and the grid-coordinate locations, a coverage metric indicating coverage of the tympanic membrane by the sensor array. However, RUPPERSBERG is concerned with analyzing “a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). For this reason, RUPPERSBERG is particularly concerned with the eardrum being within the field of view. “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]). “To reduce this risk, the ear detection device according to the present invention preferably uses data measured by the infrared sensor unit to verify that the electronic imaging unit has a free line of sight to the subject's eardrum.” ([0021]). Notably, RUPPERSBERG teaches warning the user that “the main viewing direction of the infrared sensor unit…is not directed to the eardrum and/or that there is no free line of sight to the eardrum. A corresponding warning may be emitted to the operator of the device” ([0021]). RUPPERSBERG emphasizes that automatic analysis is more desirable. ([0016]).
In the same field of endeavor, TSUBOI is primarily concerned with insuring that a tip of the thermometer is properly positioned before acquiring subsequent information. (see generally, e.g., [0102]-[0127]). “The eardrum recognition unit 221 a determines whether a tip end of the temperature sensor unit 110 faces the eardrum 14, and controls the functional units based on the determination result.” ([0122]). To this end, TSUBOI teaches determining an “eardrum occupancy rate, which is a ratio of an area in which the eardrum is shown in a captured image.” ([0113]). Like RUPPERSBERG, TSUBOI is configured to detect the eardrum. TSUBOI further teaches detecting the ear drum and determining an amount of the eardrum that occupies the image. “The eardrum recognition unit 221 a acquires a thermal image as image information including the eardrum 14 from the thermal image acquisition unit 218 and recognizes the position of the eardrum 14…The eardrum recognition unit 221 a performs the image recognition process on the acquired thermal image, and calculates a ratio occupied by the eardrum 14 (an eardrum occupancy rate) in the image..” ([0122]). “The temperature processing unit 223 determines that the temperature sensor unit 110 faces the eardrum 14 when the eardrum occupancy rate calculated by the eardrum recognition unit 221 is greater than or equal to a predetermined value, and determines that the measured temperature of the temperature measurement unit 216 is reliable.” ([0077]).
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the RUPPERSBERG device to determine, from the temperature information and the grid-coordinate locations, a coverage metric indicating coverage of the tympanic membrane by the sensor array. It would have also been obvious to configure the device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. RUPPERSBERG uses infrared data to automatically verify that the sensory array is properly positioned for imaging the eardurm. (see, e.g., [0021] and [0016]). One having ordinary skill in the art would have been motivated to determine the occupancy rate (i.e., coverage) of the tympanic membrane, as taught in TSUBOI, prior to imaging the tympanic membrane to insure there would be sufficient information for making a diagnosis. One having ordinary skill in the art would also, based on the teachings of RUPPERSBERG in view of TSUBOI, configure the device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas in order to inform the user that the device is properly positioned. There would have been a reasonable expectation of success as TSUBOI teaches that the occupancy rate can be determined using thermal images and RUPPERSBERG teaches that the device can automatically determine whether the device is properly positioned and warn the user when it is not.
RUPPERSBERG does not explicitly teach determining whether the coverage metric satisfies a coverage criterion for detecting infection areas and also determining, from the sensor data, whether an occlusion area within the ear canal impacts the coverage metric. However, as discussed above, RUPPERSBERG is particularly concerned with the eardrum being within the field of view. “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]). RUPPERSBERG teaches warning the user that “the main viewing direction of the infrared sensor unit…is not directed to the eardrum and/or that there is no free line of sight to the eardrum. A corresponding warning may be emitted to the operator of the device” ([0021]).
DOUGLAS teaches “methods and apparatuses for assisting in the acquisition and analysis of images of the tympanic membrane to provide information that may be helpful in the understanding and management of disease, such ear infection (acute otitis media).” (Abstract). Generally, DOUGLAS teaches extracting features and determining the probability those features identify the tympanic membrane. ([0014]). Similar to RUPPERSBERG, DOUGLAS is concerned with a sufficient amount of the tympanic membrane being visible and, as such, teaches guiding the user so that the tympanic membrane is sufficiently viewable. “In general, a method of guidance or an apparatus for guiding a subject to take an image may examine images (digital images) of a patient's ear canal being taken by the user, e.g., operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%,….” ([0036]). DOUGLAS is also concerned with obstructions. “Obstructions within the ear canal, including cerumen or foreign bodies may either prevent the ear exam from being completed successfully (because they partially or completely occlude the tympanic membrane) or may present a hazard during the otoscopic examination because they may be pushed by the speculum deeper into the ear.” ([0328]).
DOUGLAS teaches using a threshold to determine “when the area of the tympanic membrane (either absolute or relative to other image features) is sufficiently large…Other methods besides absolute size of the TM may be used to determine when an exam may be completed, including the sufficiently high quality capture of a particular part of the ear anatomy (e.g., the umbo of the TM), or the cone of light, or any other feature as suggested by the entity responsible for using the image or video to make a diagnosis (e.g., the physician)..” ([0327]). DOUGLAS also teaches that a machine-learning model may be trained to identify obstructions. “A cerumen/foreign object segmentation method has been created and described herein, analogous to that of TM segmentation discussed herein, to detect these obstructions.” ([0328]). If a significant obstruction exists, the user could be warned not to proceed any further. ([0304] and [0328]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas and to determine, from the sensor data, whether an occlusion area within the ear canal impacts the coverage metric. Each of RUPPERSBERG and DOUGLAS is particularly concerned with a sufficient amount of the tympanic membrane being viewable in order to obtain data that could be analyzed. Obstructions reduce the amount of the tympanic membrane that is viewable. Each of RUPPERSBERG and DOUGLAS describe instructing the user to move the device if the amount of tympanic membrane that is viewable is not sufficient. TSUBOI uses a classifier to determine an “occupancy rate” that is representation of how much of the tympanic membrane is viewable. As such, one having ordinary skill in the art would have been motivated to modify the TSUBOI classifier to not only determine which pixels correspond to the tympanic membrane but also to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. If the coverage metric does not satisfy the criterion (e.g., at least 60% of the tympanic membrane), the user could be warned or guided in moving the device, as taught in RUPPERSBERG and DOUGLAS.
One having ordinary skill in the art would have also been motivated to modify the TSUBOI classifier to determine which pixels correspond to obstructions, as taught in DOUGLAS. One would use the modified method to distinguish whether the coverage is low because of the position of the device, in which case the user could then be guided to a better position, or whether coverage is low because of the obstruction, in which case it may not be possible to guide the user to a better position and, instead, warn the user not to proceed further. (see, e.g., [0304] of DOUGLAS). One would have also wanted to identify obstructions in order to determine if the temperature data correspond to obstructions should be excluded from further analysis because obstructions have different temperatures. (see, e.g., [0021] of RUPPERSBERG). There would have been a reasonable expectation of success as DOUGLAS teaches that machine-learning models can be trained to identify obstructions.
RUPPERSBERG does not explicitly teach generating a compensated subset of the sensor data by excluding or compensating sensor data associated with the occlusion area or with portions of the sensor array that fail to satisfy the coverage criterion. However, RUPPERSBERG is concerned about obstructions and their ability to reduce useful information. (see, e.g., [0021]).
DOUGLAS describes training a model “to automatically generate a TM segmentation” of the tympanic membrane. DOUGLAS also teaches that the model could be used to exclude portions of the image that correspond to obstructions. More specifically, the segmentation model could be used “to ignore the TM portion of the image (e.g., for detection or classification of cerumen or rashes in the ear canal).” ([0177]). DOUGLAS emphasizes this later in describing a learning system that divides an image into blocks of a certain size and “regions that have certain characteristics that do not imply the presence of eardrum could be excluded automatically. The simple mean and median can now be computed for these remaining blocks.” ([0248]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to generate a compensated subset of the sensor data by excluding sensor data associated with the occlusion area. One of ordinary skill in the art would have been motivated to exclude the portions of the image that correspond to obstructions so that the subsequent analysis would only be applied to the tympanic membrane. There would have been a reasonable expectation of success as DOUGLAS teaches that machine-learning models may be trained to exclude portions of an image that do not correspond to the tympanic membrane.
RUPPERSBERG does not explicitly teach detecting, based at least in part on the compensated subset of the sensor data and the coverage metric satisfying the coverage criterion, signs of infection in the ear canal of the patient. However, RUPPERSBERG is concerned about obtaining adequate data for subsequent analysis. (see, e.g., [0021]).
DOUGLAS teaches that one must “capture a sufficiently detailed image of a tympanic membrane for use in diagnosing or analysis using the tympanic membrane.” ([0034]). DOUGLAS teaches that a minimum amount of the tympanic membrane may be necessary for analysis, e.g., “operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%, more than 35%....” ([0036], see also [0327]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to configured to detect, based at least in part on the compensated subset of the sensor data and the coverage metric satisfying the coverage criterion, signs of infection in the ear canal of the patient. One of ordinary skill in the art would have desired infrared images that include a sufficient amount of the tympanic membrane (i.e., sufficient coverage and without obstruction). As such, one would have modified the TSUBOI classifier, based on the teachings of DOUGLAS, to determine when there is a sufficient amount of the tympanic membrane in the image and to remove portions of the image that include obstructions. There would have been a reasonable expectation of success as DOUGLAS teaches machine learning models can determine when there is a sufficient amount of the tympanic membrane in the image and can remove portions of the image that include obstructions.
With respect to identifying infection areas within the ear canal based at least in part on the compensated subset of the sensor data and the coverage metric, DOUGLAS further teaches a protocol for identifying an ear ailment based on the tympanic membrane: “selecting a region of interest comprising at least a portion of the subject's tympanic membrane from a first image including at least a portion of a tympanic membrane; extracting a plurality of image features from the region of interest of the first image, wherein the image features include data derived from the color and texture data; combining the extracted features into a feature vector for the first image; applying the feature vector to a trained classification model to identify a probability of each of a plurality of different diseases….” ([0064]).
Moreover, RUPPERSBERG teaches that “[l]ocal inflammations also lead to a raise in temperature at the site of inflammation.” ([0028]). “Inflammation of the eardrum may suggest e.g. an (bacterial/viral) infection.” ([0068]). RUPPERSBERG further teaches analyzing the eardrum for “diagnosing an ear disease.” ([0133]). “[O]bjects shown in the at least on captured image may be identified (and distinguished from other objects in the subject's ear), and then the status (especially the temperature) of at least one of the identified objects is determined.” ([0133]). As previously discussed, this includes using infrared images that show a temperature distribution of the tympanic membrane. (see, e.g., [0051]-[0052])).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to be configured to identify infection areas within the ear canal based at least in part on the compensated subset of the sensor data and the coverage metric. One of ordinary skill in the art would have desired to analyze the useful information of infrared images (i.e., the regions corresponding to the tympanic membrane and not obstructions) to determine signs of infection. DOUGLAS and RUPPERSBERG both teach it is possible to detect signs of infection using image data. RUPPERSBERG specifically teaches that infrared images may be helpful in diagnosing certain conditions.
With respect to claim 2, RUPPERSBERG, as modified by DOUGLAS and TSUBOI, teaches a device that is configured determine a degree of variance as between the coverage metric and the coverage criterion, and wherein identifying the infection areas is based at least in part on the degree of variance.
RUPPERSBERG teaches that known device may incorrectly measure the temperature of the eardrum because the device may face a wall of the ear canal instead of the eardrum. ([0003]). “The reason for this is that a free line of sight from the infrared sensor unit to the eardrum or tympanic membrane is mandatory for correctly measuring the subject's body core temperature.” ([0003]).
To this end, RUPPERSBERG teaches using a wide angle video camera. “Such wide angle cameras allow detection of the subject's eardrum, even if the optical axis (‘main viewing direction’) of the camera is initially not directly centered to the eardrum. Once the eardrum has been detected in some region of the captured wide angle image, the operator of the ear inspection device may be informed, e.g. by some kind of guidance system, how to manipulate the position or orientation of the device with respect to the subject's ear so as to center the optical axis of the camera (and thus of the infrared sensor unit) to the eardrum.” Paragraph [0212] describes guiding the user in positioning the otoscope properly, which may include indicating an insertion depth, a direction of rotation, a tilting angle. In order to guide a user, the system must be comparing the current position to another more optimal position.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to determine a degree of variance as between the coverage metric and the coverage criterion, and wherein identifying the infection areas is based at least in part on the degree of variance. One would have been motivated to determine the degree of variance in order to guide the user into acquiring better data of the tympanic membrane. By acquiring better data, identification of the infection areas would necessarily be based on the degree of variance. As discussed above, the TSUBOI method (as modified by DOUGLAS) would analyze the images to determine how a user should move the device to acquire a better image. There would have been a reasonable expectation of success as RUPPERSBERG teaches that one can determine how to guide a user to a better position.
With respect to claim 3, RUPPERSBERG, as modified by DOUGLAS and TSUBOI, teaches a device that is configured to determine, from the sensor data, the occlusion area within the ear canal and determine that the occlusion area has impacted the coverage metric or an ability to detect the signs of the infection and output an alert based at least in part on determination that the occlusion area has impacted the coverage metric or the ability to detect the signs of the infection. As discussed above, the TSUBOI method (as modified by DOUGLAS) would identify the occlusion areas (e.g., obstructions). Each of RUPPERSBERG and DOUGLAS describe instructing the user to move the device if the amount of tympanic membrane that is viewable is not sufficient. TSUBOI uses a classifier to determine an “occupancy rate” that is representation of how much of the tympanic membrane is viewable. As such, one having ordinary skill in the art would have been motivated to modify the TSUBOI classifier to not only determine which pixels correspond to the tympanic membrane but also to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. If the coverage metric does not satisfy the criterion (e.g., at least 60% of the tympanic membrane), the user could be warned or guided in moving the device, as taught in RUPPERSBERG and DOUGLAS.
With respect to claim 5, RUPPERSBERG does not explicitly teach monitoring, over a period of time, temperature readings from the sensor array; determining, from the temperature readings received over the period of time, that the infection has worsened; and providing a notification that the infection has worsened.
DOUGLAS teaches various embodiments for acquiring images of the tympanic membrane. ([0014]). DOUGLAS’s embodiments may configure “for use with a home or clinical device that includes an otoscope (e.g., speculum, lens/lenses, and video/image capture capability). Images may be acquired until the method/apparatus indicates, e.g., visually or audibly, that an adequate image has been taken. The image(s) may then be stored, transmitted, and/or analyzed. For example, stored images may be transmitted to a medical provider for further analysis, or to a third-party analysis center.” ([0034]).
DOUGLAS also teaches tracking treatment progress. “Once a diagnosis is made and an intervention is prescribed, it is helpful to track the progression of the disease via continuous exams and to suggest altering interventions if the disease is progressing or otherwise not improving as rapidly as expected. Methods and software tools can be a powerful aid in this tracking.” ([0265]). “A method for continuous monitoring is shown in FIG. 20. Initially, an exam is performed, a diagnosis is made (possibly with help from the automated diagnosis method, described herein) and an intervention (including, possibly, no intervention or “watchful waiting”) is prescribed (possibly with help from the automated prognosis method). After waiting for some period, e.g., one day, the exam is repeated. If the disease is improving as expected (possibly as determined with help from the automated method to determine disease stage) but has not yet resolved, the waiting period plus exam cycle may be repeated. If the disease is not improving as expected, a new intervention is prescribed based on the results of the most recent exam and, possibly, prior exams (again, possibly with help from the automated prognosis method).” ([0266]). By recommending a new intervention using the automated prognosis method, the user would be notified that the infection has worsened.
It would have been obvious to one having ordinary skill in the art at the time of filing to monitor, over a period of time, temperature readings from the sensor array, determine that the infection has worsened, and provide a notification that the infection has worsened, as taught in DOUGLAS. One of ordinary skill in the art would have been motivated to configured the RUPPERSBERG device to monitor and notify the user about the progression of the infection so that the user would know if the intervention is working. There would have been a reasonable expectation of success as DOUGLAS teaches that one can monitor the progression of an ear infection and keep the user updated on the progress.
With respect to claim 6, RUPPERSBERG teaches determining heat distribution asymmetries as between first temperature readings of a first ear of the patient and second temperature readings of a second ear of the patient, and wherein detecting the infection is based at least in part on comparative diagnosis performed on the first temperature readings and the second temperature readings. More specifically, RUPPERSBERG teaches that “[i]f an elevated temperature (i.e. a temperature above the normal body core temperature of a human being) is detected by the infrared sensor unit when the ear inspection device according to the present invention is introduced at least partially in one of the two exterior ear canals of the subject, this does not always allow to conclude that the subject has an elevated body core temperature, i.e. fever. Instead, the measured elevated temperature may result from a local inflammation of the eardrum of the ear into which the device has been introduced. Local inflammations also lead to a raise in temperature at the site of inflammation. To distinguish between these two cases, i.e. fever vs. local inflammation, it is advantageous to subsequently carry out the temperature measurement at both (i.e. left and right) ears of the subject.” ([0028]).
With respect to claim 8, RUPPERSBERG teaches a device for detecting ear infections (Title and Abstract). Although described in the context of determine a body core temperature, the RUPPERSBERG is also described as being capable of “diagnosing an ear disease,” which may include “reliably diagnosing e.g. an inflammation of the eardrum without the need of assistance of a skilled physician.” ([0133]). The device comprising:
one or more processors. Claim 3 recites “a logic unit configured for receiving and processing signals from the infrared sensor unit….”.
a sensor array configured to measure tympanic temperatures across multiple points on a tympanic membrane of a patient. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). RUPPERSBERG also describes technology enabling thermal images of sufficient resolution. “Therefore, wafer-level imaging technology allows obtaining images (of the temperature distribution and/or of light in the visual range) of “sufficient” resolution of the eardrum, e.g. images of 250 pixels×250 pixels, with a footprint of the camera (including a lens) of only about 1 mm×1 mm or even smaller.” ([0052]).
a housing configured to contain the one or more processors and the sensor array, the housing having a distal end configured for at least partial insertion into an ear canal of the patient. Figure 1 of RUPPERSBERG shows an ear inspection device 10 having a head portion 14 that includes a logic unit 44. The head portion 14 includes a distal end 18 that “is adapted to be introduced into a subject's ear canal.” ([0155]).
receiving an indication that a portion of the device has been inserted into an ear canal of the patient. RUPPERSBERG describes a “moving mechanism” that is triggered when the device is inserted into the ear canal. “As soon as the probe cover 60 gets in contact with an inner lateral surface of the ear canal, a friction force is exerted on the probe cover 60. The friction force depends on the position of the head portion 14 within the ear canal: the friction force increases with increasing insertion depth.” ([0193]). The moving mechanism is electrically connected to one of the cameras. “In case the probe cover 60 is axially displaced, the motion detector can emit an electric signal which is transmitted to the at least one camera 40.1 or any logical unit or control unit, evoking start-up or powering of the camera 40.1. In such a way, by means of motion detection or detection of the axial position of the probe cover 60, the camera 40.1 can be powered at a time when the camera 40.1 is in visual communication with the eardrum.” ([0199]). Notably, the imaging unit and infrared sensor unit operate simultaneously. ([0165]).
receiving sensor data from the sensor array, the sensor data indicating temperature information at grid-coordinate locations of the sensor array. The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]).
RUPPERSBERG does not explicitly teach that the device includes a non-transitory computer-readable media, housed in the housing, storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. Nonetheless, RUPPERSBERG a logic unit 44 that processes infrared/imaging data and is configured to evaluate the images. (see, e.g., claims 3 and 4).
In the same field of endeavor, TSUBOI teaches “a type of clinical thermometer that measures a body temperature by measuring radiant heat emitted from an eardrum….” ([0002]). TSUBOI teaches that it is challenging to acquire measurements of the eardrum continuously or over a longer period of time due to the thermometer being incorrectly positioned. ([0004]-[0005]). To address these challenges, TSUBOI teaches “an eardrum recognition unit configured to recognize a position of an eardrum based on image information regarding the eardrum, a temperature measurement unit configured to acquire a temperature within an external ear canal including the eardrum, and a temperature processing unit configured to determine a temperature of the eardrum based on a recognition result of the eardrum recognition unit and a measured temperature of the temperature measurement unit.” ([0007]). TSUBOI teaches a computer program that can cause a computer to perform the operations of these processing units. ([0009], [0131], [0132]).
It would have been obvious to one having ordinary skill in the art at the time of filing to add a non-transitory computer-readable media to the RUPPERSBERG device, housed in the housing and storing instructions, as recited. One of ordinary skill in the art could have added a non-transitory computer-readable medium within the housing using known methods. One of ordinary skill in the art would have recognized that the results of adding the computer-readable medium would be predictable.
RUPPERSBERG does not explicitly teach that the device is configured to determine, from the temperature information and the grid-coordinate locations, a coverage metric indicating coverage of the tympanic membrane by the sensor array. However, RUPPERSBERG is concerned with analyzing “a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). For this reason, RUPPERSBERG is particularly concerned with the eardrum being within the field of view. “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]). “To reduce this risk, the ear detection device according to the present invention preferably uses data measured by the infrared sensor unit to verify that the electronic imaging unit has a free line of sight to the subject's eardrum.” ([0021]). Notably, RUPPERSBERG teaches warning the user that “the main viewing direction of the infrared sensor unit…is not directed to the eardrum and/or that there is no free line of sight to the eardrum. A corresponding warning may be emitted to the operator of the device” ([0021]). RUPPERSBERG emphasizes that automatic analysis is more desirable. ([0016]).
In the same field of endeavor, TSUBOI is primarily concerned with insuring that a tip of the thermometer is properly positioned before acquiring subsequent information. (see generally, e.g., [0102]-[0127]). “The eardrum recognition unit 221 a determines whether a tip end of the temperature sensor unit 110 faces the eardrum 14, and controls the functional units based on the determination result.” ([0122]). To this end, TSUBOI teaches determining an “eardrum occupancy rate, which is a ratio of an area in which the eardrum is shown in a captured image.” ([0113]). Like RUPPERSBERG, TSUBOI is configured to detect the eardrum. TSUBOI further teaches detecting the ear drum and determining an amount of the eardrum that occupies the image. “The eardrum recognition unit 221 a acquires a thermal image as image information including the eardrum 14 from the thermal image acquisition unit 218 and recognizes the position of the eardrum 14…The eardrum recognition unit 221 a performs the image recognition process on the acquired thermal image, and calculates a ratio occupied by the eardrum 14 (an eardrum occupancy rate) in the image..” ([0122]). “The temperature processing unit 223 determines that the temperature sensor unit 110 faces the eardrum 14 when the eardrum occupancy rate calculated by the eardrum recognition unit 221 is greater than or equal to a predetermined value, and determines that the measured temperature of the temperature measurement unit 216 is reliable.” ([0077]).
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the RUPPERSBERG device to determine, from the temperature information and the grid-coordinate locations, a coverage metric indicating coverage of the tympanic membrane by the sensor array. It would have also been obvious to configure the device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. RUPPERSBERG uses infrared data to automatically verify that the sensory array is properly positioned for imaging the eardurm. (see, e.g., [0021] and [0016]). One having ordinary skill in the art would have been motivated to determine the occupancy rate (i.e., coverage) of the tympanic membrane, as taught in TSUBOI, prior to imaging the tympanic membrane to insure there would be sufficient information for making a diagnosis. One having ordinary skill in the art would also, based on the teachings of RUPPERSBERG in view of TSUBOI, configure the device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas in order to inform the user that the device is properly positioned. There would have been a reasonable expectation of success as TSUBOI teaches that the occupancy rate can be determined using thermal images and RUPPERSBERG teaches that the device can automatically determine whether the device is properly positioned and warn the user when it is not.
RUPPERSBERG does not explicitly teach determining whether the coverage metric satisfies a coverage criterion for reducing positional inaccuracy. However, as discussed above, RUPPERSBERG is particularly concerned with the eardrum being within the field of view.
TSUBOI teaches determining whether the coverage metric satisfies a coverage criterion for reducing positional inaccuracy. ([0079] and [0102]-[0103]). “The notification unit 225 notifies the user of a position state of the temperature sensor unit 110, which is a temperature measurement unit for the eardrum 14, that is, whether the tip end of the temperature sensor unit 110 faces the eardrum 14, based on the detection result of the eardrum recognition unit 221.” ([0079]). “It is possible to determine whether the temperature sensor unit 110 substantially directly faces the eardrum 14 and temperature measurement is accurately performed from the eardrum occupancy rate.” ([0102]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the device to be configured to determine whether the coverage metric satisfies a coverage criterion for reducing positional inaccuracy, as taught in TSUBOI. One of ordinary skill in the art would have been motivated to determine whether the device is sufficiently facing the eardrum, as taught in TSUBOI. If the coverage metric is unsatisfactory, the device would direct the user to adjust and find a better position for the probe. There would have been a reasonable expectation of success as RUPPERSBERG and TSUBOI teach guiding the user to obtain better quality information.
RUPPERSBERG does not explicitly teach determining, from the sensor data, whether an occlusion area within the ear canal impacts the coverage metric. However, as discussed above, RUPPERSBERG is particularly concerned with the eardrum being within the field of view. “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]). RUPPERSBERG teaches warning the user that “the main viewing direction of the infrared sensor unit…is not directed to the eardrum and/or that there is no free line of sight to the eardrum. A corresponding warning may be emitted to the operator of the device” ([0021]).
DOUGLAS teaches “methods and apparatuses for assisting in the acquisition and analysis of images of the tympanic membrane to provide information that may be helpful in the understanding and management of disease, such ear infection (acute otitis media).” (Abstract). Generally, DOUGLAS teaches extracting features and determining the probability those features identify the tympanic membrane. ([0014]). Similar to RUPPERSBERG, DOUGLAS is concerned with a sufficient amount of the tympanic membrane being visible and, as such, teaches guiding the user so that the tympanic membrane is sufficiently viewable. “In general, a method of guidance or an apparatus for guiding a subject to take an image may examine images (digital images) of a patient's ear canal being taken by the user, e.g., operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%,….” ([0036]). DOUGLAS is also concerned with obstructions. “Obstructions within the ear canal, including cerumen or foreign bodies may either prevent the ear exam from being completed successfully (because they partially or completely occlude the tympanic membrane) or may present a hazard during the otoscopic examination because they may be pushed by the speculum deeper into the ear.” ([0328]).
DOUGLAS teaches using a threshold to determine “when the area of the tympanic membrane (either absolute or relative to other image features) is sufficiently large…Other methods besides absolute size of the TM may be used to determine when an exam may be completed, including the sufficiently high quality capture of a particular part of the ear anatomy (e.g., the umbo of the TM), or the cone of light, or any other feature as suggested by the entity responsible for using the image or video to make a diagnosis (e.g., the physician)..” ([0327]). DOUGLAS also teaches that a machine-learning model may be trained to identify obstructions. “A cerumen/foreign object segmentation method has been created and described herein, analogous to that of TM segmentation discussed herein, to detect these obstructions.” ([0328]). If a significant obstruction exists, the user could be warned not to proceed any further. ([0304] and [0328]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas and to determine, from the sensor data, whether an occlusion area within the ear canal impacts the coverage metric. Each of RUPPERSBERG and DOUGLAS is particularly concerned with a sufficient amount of the tympanic membrane being viewable in order to obtain data that could be analyzed. Obstructions reduce the amount of the tympanic membrane that is viewable. Each of RUPPERSBERG and DOUGLAS describe instructing the user to move the device if the amount of tympanic membrane that is viewable is not sufficient. TSUBOI uses a classifier to determine an “occupancy rate” that is representation of how much of the tympanic membrane is viewable. As such, one having ordinary skill in the art would have been motivated to modify the TSUBOI classifier to not only determine which pixels correspond to the tympanic membrane but also to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. If the coverage metric does not satisfy the criterion (e.g., at least 60% of the tympanic membrane), the user could be warned or guided in moving the device, as taught in RUPPERSBERG and DOUGLAS.
One having ordinary skill in the art would have also been motivated to modify the TSUBOI classifier to determine which pixels correspond to obstructions, as taught in DOUGLAS. One would use the modified method to distinguish whether the coverage is low because of the position of the device, in which case the user could then be guided to a better position, or whether coverage is low because of the obstruction, in which case it may not be possible to guide the user to a better position and, instead, warn the user not to proceed further. (see, e.g., [0304] of DOUGLAS). One would have also wanted to identify obstructions in order to determine if the temperature data correspond to obstructions should be excluded from further analysis because obstructions have different temperatures. (see, e.g., [0021] of RUPPERSBERG). There would have been a reasonable expectation of success as DOUGLAS teaches that machine-learning models can be trained to identify obstructions.
RUPPERSBERG does not explicitly teach generating a compensated subset of the sensor data by excluding or compensating sensor data associated with the occlusion area or with portions of the sensor array that fail to satisfy the coverage criterion. However, RUPPERSBERG is concerned about obstructions and their ability to reduce useful information. (see, e.g., [0021]).
DOUGLAS describes training a model “to automatically generate a TM segmentation” of the tympanic membrane. DOUGLAS also teaches that the model could be used to exclude portions of the image that correspond to obstructions. More specifically, the segmentation model could be used “to ignore the TM portion of the image (e.g., for detection or classification of cerumen or rashes in the ear canal).” ([0177]). DOUGLAS emphasizes this later in describing a learning system that divides an image into blocks of a certain size and “regions that have certain characteristics that do not imply the presence of eardrum could be excluded automatically. The simple mean and median can now be computed for these remaining blocks.” ([0248]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to generate a compensated subset of the sensor data by excluding sensor data associated with the occlusion area. One of ordinary skill in the art would have been motivated to exclude the portions of the image that correspond to obstructions so that the subsequent analysis would only be applied to the tympanic membrane. There would have been a reasonable expectation of success as DOUGLAS teaches that machine-learning models may be trained to exclude portions of an image that do not correspond to the tympanic membrane.
RUPPERSBERG does not explicitly teach detecting, based at least in part on the compensated subset of the sensor data and the coverage metric satisfying the coverage criterion, signs of infection in the ear canal of the patient. However, RUPPERSBERG is concerned about obtaining adequate data for subsequent analysis. (see, e.g., [0021]).
DOUGLAS teaches that one must “capture a sufficiently detailed image of a tympanic membrane for use in diagnosing or analysis using the tympanic membrane.” ([0034]). DOUGLAS teaches that a minimum amount of the tympanic membrane may be necessary for analysis, e.g., “operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%, more than 35%....” ([0036], see also [0327]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to configured to detect, based at least in part on the compensated subset of the sensor data and the coverage metric satisfying the coverage criterion, signs of infection in the ear canal of the patient. One of ordinary skill in the art would have desired infrared images that include a sufficient amount of the tympanic membrane (i.e., sufficient coverage and without obstruction). As such, one would have modified the TSUBOI classifier, based on the teachings of DOUGLAS, to determine when there is a sufficient amount of the tympanic membrane in the image and to remove portions of the image that include obstructions. There would have been a reasonable expectation of success as DOUGLAS teaches machine learning models can determine when there is a sufficient amount of the tympanic membrane in the image and can remove portions of the image that include obstructions.
With respect to identifying infection areas within the ear canal based at least in part on the compensated subset of the sensor data and the coverage metric, DOUGLAS further teaches a protocol for identifying an ear ailment based on the tympanic membrane: “selecting a region of interest comprising at least a portion of the subject's tympanic membrane from a first image including at least a portion of a tympanic membrane; extracting a plurality of image features from the region of interest of the first image, wherein the image features include data derived from the color and texture data; combining the extracted features into a feature vector for the first image; applying the feature vector to a trained classification model to identify a probability of each of a plurality of different diseases….” ([0064]).
Moreover, RUPPERSBERG teaches that “[l]ocal inflammations also lead to a raise in temperature at the site of inflammation.” ([0028]). “Inflammation of the eardrum may suggest e.g. an (bacterial/viral) infection.” ([0068]). RUPPERSBERG further teaches analyzing the eardrum for “diagnosing an ear disease.” ([0133]). “[O]bjects shown in the at least on captured image may be identified (and distinguished from other objects in the subject's ear), and then the status (especially the temperature) of at least one of the identified objects is determined.” ([0133]). As previously discussed, this includes using infrared images that show a temperature distribution of the tympanic membrane. (see, e.g., [0051]-[0052])).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to be configured to identify infection areas within the ear canal based at least in part on the compensated subset of the sensor data and the coverage metric. One of ordinary skill in the art would have desired to analyze the useful information of infrared images (i.e., the regions corresponding to the tympanic membrane and not obstructions) to determine signs of infection. DOUGLAS and RUPPERSBERG both teach it is possible to detect signs of infection using image data. RUPPERSBERG specifically teaches that infrared images may be helpful in diagnosing certain conditions.
With respect to claim 9, RUPPERSBERG teaches wherein the sensor array comprises a grid-based infrared temperature sensor. “The infrared sensor unit of the ear inspection device according to the present invention may comprise a plurality of infrared sensor elements for detecting infrared radiation from different regions of the ear.” ([0050]). “This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). “Therefore, wafer-level imaging technology allows obtaining images (of the temperature distribution and/or of light in the visual range) of “sufficient” resolution of the eardrum, e.g. images of 250 pixels×250 pixels, with a footprint of the camera (including a lens) of only about 1 mm×1 mm or even smaller.” ([0052]).
With respect to claim 10, RUPPERSBERG, as modified by TSUBOI and DOUGLAS teaches outputting an alert when the coverage metric does not satisfy the coverage criterion for accurately detecting the infection. RUPPERSBERG teaches warning the user that “the main viewing direction of the infrared sensor unit…is not directed to the eardrum and/or that there is no free line of sight to the eardrum. A corresponding warning may be emitted to the operator of the device” ([0021]). DOUGLAS also teaches guiding the user. “In general, a method of guidance or an apparatus for guiding a subject to take an image may examine images (digital images) of a patient's ear canal being taken by the user, e.g., operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%,….” ([0036]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to output an alert when the coverage metric does not satisfy the coverage criterion for accurately detecting the infection. One of ordinary skill in the art would have been motivated to alert the user so that the user could then move the device to obtain a better image. There would have been a reasonable expectation of success as RUPPERSBERG and DOUGLAS teaches a user can be guided to obtain a better image.
With respect to claim 11, RUPPERSBERG, as modified by TSUBOI and DOUGLAS teaches determining a subset of the temperature information to utilize based at least in part on the coverage metric and whether the occlusion area impacts the coverage metric, and wherein identifying the infection areas is based at least in part on the subset of the temperature information.
As discussed above in the rejection of claim 8, RUPPERSBERG teaches that obstructions can distort thermal images because the obstructions have a different temperature. “Normally, the temperature at the surface of the eardrum (substantially corresponding to the subject's body core temperature) is higher than the temperature…of earwax, hair or dirt in the exterior ear canal. Therefore, if the infrared sensor unit measures temperature values significantly (e.g. more than 2° C.) below the normal body core temperature of a human being….” ([0021]).
DOUGLAS teaches identifying obstructions within the image and excluding those portions of the image that correspond to the obstructions. DOUGLAS describes training a model “to automatically generate a TM segmentation” of the tympanic membrane. DOUGLAS also teaches that the model could be used to exclude portions of the image that correspond to obstructions. More specifically, the segmentation model could be used “to ignore the TM portion of the image (e.g., for detection or classification of cerumen or rashes in the ear canal).” ([0177]). DOUGLAS emphasizes this later in describing a learning system that divides an image into blocks of a certain size and “regions that have certain characteristics that do not imply the presence of eardrum could be excluded automatically. The simple mean and median can now be computed for these remaining blocks.” ([0248]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to determine a subset of the temperature information to utilize based at least in part on the coverage metric and whether the occlusion area impacts the coverage metric and to identify the infection areas is based at least in part on the subset of the temperature information. One of ordinary skill in the art would have been motivated to exclude the portions of the image that correspond to obstructions so that the subsequent analysis would only be applied to the tympanic membrane. There would have been a reasonable expectation of success as DOUGLAS teaches that machine-learning models may be trained to exclude portions of an image that do not correspond to the tympanic membrane.
With respect to claim 12, RUPPERSBERG teaches using a display unit (see, e.g., [0016]) but does not explicitly teach displaying a representation of the coverage metric of the sensor array relative to the tympanic membrane.
However, DOUGLAS teaches guiding the user to properly position the device so that better quality images may be obtained. DOUGLAS teaches “A method of guiding a subject using an otoscope coupled to a display device to image a tympanic membrane may include:…displaying, on the display device, the image;…indicating if the ear canal is occluded; indicating on the display device, a direction to orient the otoscope based on the detected one or more deeper regions….” ([0042]). DOUGLAS further teaches displaying the images of the TM acquiring images when a sufficient portion is shown. “[T]he apparatus may be configured to examine the images (or a subset of the images) being received and/or displayed, to identify a TM or a portion of a TM, as described herein. The apparatus may be configured to determine directionality (e.g., to center the TM) based on the position of the identified probable TM region on the screen, and provide indicators (e.g., arrows, icons, audible instructions/guidance) to guide the subject in positioning the otoscope. Additionally or alternatively, the apparatus may automatically take one or more images when a substantial portion (e.g., above a threshold of the TM or percentable of the image field of view) is present.”
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the device to display a representation of the coverage metric of the sensor array relative to the tympanic membrane. One of ordinary skill in the art would have motivated to display a representation of the coverage metric so that a user could be informed as to how the coverage metric changes with adjusting a position of the device. There would have been a reasonable expectation of success as DOUGLAS teaches that the display unit can guide a user.
With respect to claim 13, RUPPERSBERG does not explicitly teaches displaying a representation of the infection areas via the display unit. However, RUPPERSBERG teaches displaying images to the user using a display unit. ([0016] and [0162]). DOUGLAS teaches “[a]fter the automated diagnosis method has run, each possible disease and diagnosis score will be displayed, possibly alongside a representative exam image for that disease.”
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to display a representation of the infection areas via the display unit. One of ordinary skill in the art would have been motivated to display the representation of the infection areas so that a doctor or user could view images of the diseased tympanic membrane.
With respect to claim 14, RUPPERSBERG teaches displaying a representation of the infection areas via the display unit. RUPPERSBERG teaches that “[i]f an elevated temperature (i.e. a temperature above the normal body core temperature of a human being) is detected by the infrared sensor unit when the ear inspection device according to the present invention is introduced at least partially in one of the two exterior ear canals of the subject, this does not always allow to conclude that the subject has an elevated body core temperature, i.e. fever. Instead, the measured elevated temperature may result from a local inflammation of the eardrum of the ear into which the device has been introduced. Local inflammations also lead to a raise in temperature at the site of inflammation. To distinguish between these two cases, i.e. fever vs. local inflammation, it is advantageous to subsequently carry out the temperature measurement at both (i.e. left and right) ears of the subject.” ([0028]).
It would have been obvious to one having ordinary skill in the art at the time of filing to repeat the RUPPERSBERG-TSUBOI-DOUGLAS protocol as described above with respect to claim 8. More specifically, one having ordinary skill in the art would acquire infrared images of the left ear and then repeat the same process to acquire infrared images of the right ear in order to determine an ear infection as taught by RUPPERSBERG.
With respect to claim 15, RUPPERSBERG teaches a method for detecting ear infections (see rejection of claim 1 above), the method comprising:
receiving an indication that a portion of a device for measuring inner ear temperatures has been inserted into an ear canal of a patient. RUPPERSBERG describes a “moving mechanism” that is triggered when the device is inserted into the ear canal. “As soon as the probe cover 60 gets in contact with an inner lateral surface of the ear canal, a friction force is exerted on the probe cover 60. The friction force depends on the position of the head portion 14 within the ear canal: the friction force increases with increasing insertion depth.” ([0193]). The moving mechanism is electrically connected to one of the cameras. “In case the probe cover 60 is axially displaced, the motion detector can emit an electric signal which is transmitted to the at least one camera 40.1 or any logical unit or control unit, evoking start-up or powering of the camera 40.1. In such a way, by means of motion detection or detection of the axial position of the probe cover 60, the camera 40.1 can be powered at a time when the camera 40.1 is in visual communication with the eardrum.” ([0199]). Notably, the imaging unit and infrared sensor unit operate simultaneously. ([0165]).
receiving sensor data from the sensor array, the sensor data indicating temperature information at grid-coordinate locations of the sensor array. The infrared sensor unit may comprise “an infrared camera configured for capturing images based on radiation in the infrared range from the subject's ear. This allows for obtaining a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]).
RUPPERSBERG does not explicitly teach that the method includes determining, from the temperature information and the grid-coordinate locations, a coverage metric indicating coverage of the tympanic membrane by the sensor array. However, RUPPERSBERG is concerned with analyzing “a two-dimensional image of the temperature distribution in the area observed by the infrared camera.” ([0051]). For this reason, RUPPERSBERG is particularly concerned with the eardrum being within the field of view. “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]). “To reduce this risk, the ear detection device according to the present invention preferably uses data measured by the infrared sensor unit to verify that the electronic imaging unit has a free line of sight to the subject's eardrum.” ([0021]). Notably, RUPPERSBERG teaches warning the user that “the main viewing direction of the infrared sensor unit…is not directed to the eardrum and/or that there is no free line of sight to the eardrum. A corresponding warning may be emitted to the operator of the device” ([0021]). RUPPERSBERG emphasizes that automatic analysis is more desirable. ([0016]).
In the same field of endeavor, TSUBOI is primarily concerned with insuring that a tip of the thermometer is properly positioned before acquiring subsequent information. (see generally, e.g., [0102]-[0127]). “The eardrum recognition unit 221 a determines whether a tip end of the temperature sensor unit 110 faces the eardrum 14, and controls the functional units based on the determination result.” ([0122]). To this end, TSUBOI teaches determining an “eardrum occupancy rate, which is a ratio of an area in which the eardrum is shown in a captured image.” ([0113]). Like RUPPERSBERG, TSUBOI is configured to detect the eardrum. TSUBOI further teaches detecting the ear drum and determining an amount of the eardrum that occupies the image. “The eardrum recognition unit 221 a acquires a thermal image as image information including the eardrum 14 from the thermal image acquisition unit 218 and recognizes the position of the eardrum 14…The eardrum recognition unit 221 a performs the image recognition process on the acquired thermal image, and calculates a ratio occupied by the eardrum 14 (an eardrum occupancy rate) in the image..” ([0122]). “The temperature processing unit 223 determines that the temperature sensor unit 110 faces the eardrum 14 when the eardrum occupancy rate calculated by the eardrum recognition unit 221 is greater than or equal to a predetermined value, and determines that the measured temperature of the temperature measurement unit 216 is reliable.” ([0077]).
It would have been obvious to one having ordinary skill in the art at the time of filing to configure the RUPPERSBERG device to determine, from the temperature information and the grid-coordinate locations, a coverage metric indicating coverage of the tympanic membrane by the sensor array. It would have also been obvious to configure the device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. RUPPERSBERG uses infrared data to automatically verify that the sensory array is properly positioned for imaging the eardurm. (see, e.g., [0021] and [0016]). One having ordinary skill in the art would have been motivated to determine the occupancy rate (i.e., coverage) of the tympanic membrane, as taught in TSUBOI, prior to imaging the tympanic membrane to insure there would be sufficient information for making a diagnosis. One having ordinary skill in the art would also, based on the teachings of RUPPERSBERG in view of TSUBOI, configure the device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas in order to inform the user that the device is properly positioned. There would have been a reasonable expectation of success as TSUBOI teaches that the occupancy rate can be determined using thermal images and RUPPERSBERG teaches that the device can automatically determine whether the device is properly positioned and warn the user when it is not.
RUPPERSBERG does not explicitly teach determining whether the coverage metric satisfies a coverage criterion for detecting infection areas and also determining, from the sensor data, whether an occlusion area within the ear canal impacts the coverage metric. However, as discussed above, RUPPERSBERG is particularly concerned with the eardrum being within the field of view. “When the ear inspection device according to the present invention is used as otoscope, there is a certain risk—especially if the operator is a lay person—that the images captured by the electronic imaging unit do not show the eardrum, but instead portions of the wall of the exterior ear canal and/or earwax, hair or dirt blocking the exterior ear canal and, thus, the free view on the eardrum.” ([0021]). RUPPERSBERG teaches warning the user that “the main viewing direction of the infrared sensor unit…is not directed to the eardrum and/or that there is no free line of sight to the eardrum. A corresponding warning may be emitted to the operator of the device” ([0021]).
DOUGLAS teaches “methods and apparatuses for assisting in the acquisition and analysis of images of the tympanic membrane to provide information that may be helpful in the understanding and management of disease, such ear infection (acute otitis media).” (Abstract). Generally, DOUGLAS teaches extracting features and determining the probability those features identify the tympanic membrane. ([0014]). Similar to RUPPERSBERG, DOUGLAS is concerned with a sufficient amount of the tympanic membrane being visible and, as such, teaches guiding the user so that the tympanic membrane is sufficiently viewable. “In general, a method of guidance or an apparatus for guiding a subject to take an image may examine images (digital images) of a patient's ear canal being taken by the user, e.g., operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%,….” ([0036]). DOUGLAS is also concerned with obstructions. “Obstructions within the ear canal, including cerumen or foreign bodies may either prevent the ear exam from being completed successfully (because they partially or completely occlude the tympanic membrane) or may present a hazard during the otoscopic examination because they may be pushed by the speculum deeper into the ear.” ([0328]).
DOUGLAS teaches using a threshold to determine “when the area of the tympanic membrane (either absolute or relative to other image features) is sufficiently large…Other methods besides absolute size of the TM may be used to determine when an exam may be completed, including the sufficiently high quality capture of a particular part of the ear anatomy (e.g., the umbo of the TM), or the cone of light, or any other feature as suggested by the entity responsible for using the image or video to make a diagnosis (e.g., the physician)..” ([0327]). DOUGLAS also teaches that a machine-learning model may be trained to identify obstructions. “A cerumen/foreign object segmentation method has been created and described herein, analogous to that of TM segmentation discussed herein, to detect these obstructions.” ([0328]). If a significant obstruction exists, the user could be warned not to proceed any further. ([0304] and [0328]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas and to determine, from the sensor data, whether an occlusion area within the ear canal impacts the coverage metric. Each of RUPPERSBERG and DOUGLAS is particularly concerned with a sufficient amount of the tympanic membrane being viewable in order to obtain data that could be analyzed. Obstructions reduce the amount of the tympanic membrane that is viewable. Each of RUPPERSBERG and DOUGLAS describe instructing the user to move the device if the amount of tympanic membrane that is viewable is not sufficient. TSUBOI uses a classifier to determine an “occupancy rate” that is representation of how much of the tympanic membrane is viewable. As such, one having ordinary skill in the art would have been motivated to modify the TSUBOI classifier to not only determine which pixels correspond to the tympanic membrane but also to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. If the coverage metric does not satisfy the criterion (e.g., at least 60% of the tympanic membrane), the user could be warned or guided in moving the device, as taught in RUPPERSBERG and DOUGLAS.
One having ordinary skill in the art would have also been motivated to modify the TSUBOI classifier to determine which pixels correspond to obstructions, as taught in DOUGLAS. One would use the modified method to distinguish whether the coverage is low because of the position of the device, in which case the user could then be guided to a better position, or whether coverage is low because of the obstruction, in which case it may not be possible to guide the user to a better position and, instead, warn the user not to proceed further. (see, e.g., [0304] of DOUGLAS). One would have also wanted to identify obstructions in order to determine if the temperature data correspond to obstructions should be excluded from further analysis because obstructions have different temperatures. (see, e.g., [0021] of RUPPERSBERG). There would have been a reasonable expectation of success as DOUGLAS teaches that machine-learning models can be trained to identify obstructions.
RUPPERSBERG does not explicitly teach generating a compensated subset of the sensor data by excluding or compensating sensor data associated with the occlusion area or with portions of the sensor array that fail to satisfy the coverage criterion. However, RUPPERSBERG is concerned about obstructions and their ability to reduce useful information. (see, e.g., [0021]).
DOUGLAS describes training a model “to automatically generate a TM segmentation” of the tympanic membrane. DOUGLAS also teaches that the model could be used to exclude portions of the image that correspond to obstructions. More specifically, the segmentation model could be used “to ignore the TM portion of the image (e.g., for detection or classification of cerumen or rashes in the ear canal).” ([0177]). DOUGLAS emphasizes this later in describing a learning system that divides an image into blocks of a certain size and “regions that have certain characteristics that do not imply the presence of eardrum could be excluded automatically. The simple mean and median can now be computed for these remaining blocks.” ([0248]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to generate a compensated subset of the sensor data by excluding sensor data associated with the occlusion area. One of ordinary skill in the art would have been motivated to exclude the portions of the image that correspond to obstructions so that the subsequent analysis would only be applied to the tympanic membrane. There would have been a reasonable expectation of success as DOUGLAS teaches that machine-learning models may be trained to exclude portions of an image that do not correspond to the tympanic membrane.
RUPPERSBERG does not explicitly teach detecting, based at least in part on the compensated subset of the sensor data and the coverage metric satisfying the coverage criterion, signs of infection in the ear canal of the patient. However, RUPPERSBERG is concerned about obtaining adequate data for subsequent analysis. (see, e.g., [0021]).
DOUGLAS teaches that one must “capture a sufficiently detailed image of a tympanic membrane for use in diagnosing or analysis using the tympanic membrane.” ([0034]). DOUGLAS teaches that a minimum amount of the tympanic membrane may be necessary for analysis, e.g., “operating an otoscope to determine when a minimum amount of tympanic membrane is showing (e.g., more than 20%, more than 25%, more than 30%, more than 35%....” ([0036], see also [0327]).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to configured to detect, based at least in part on the compensated subset of the sensor data and the coverage metric satisfying the coverage criterion, signs of infection in the ear canal of the patient. One of ordinary skill in the art would have desired infrared images that include a sufficient amount of the tympanic membrane (i.e., sufficient coverage and without obstruction). As such, one would have modified the TSUBOI classifier, based on the teachings of DOUGLAS, to determine when there is a sufficient amount of the tympanic membrane in the image and to remove portions of the image that include obstructions. There would have been a reasonable expectation of success as DOUGLAS teaches machine learning models can determine when there is a sufficient amount of the tympanic membrane in the image and can remove portions of the image that include obstructions.
With respect to identifying infection areas within the ear canal based at least in part on the compensated subset of the sensor data and the coverage metric, DOUGLAS further teaches a protocol for identifying an ear ailment based on the tympanic membrane: “selecting a region of interest comprising at least a portion of the subject's tympanic membrane from a first image including at least a portion of a tympanic membrane; extracting a plurality of image features from the region of interest of the first image, wherein the image features include data derived from the color and texture data; combining the extracted features into a feature vector for the first image; applying the feature vector to a trained classification model to identify a probability of each of a plurality of different diseases….” ([0064]).
Moreover, RUPPERSBERG teaches that “[l]ocal inflammations also lead to a raise in temperature at the site of inflammation.” ([0028]). “Inflammation of the eardrum may suggest e.g. an (bacterial/viral) infection.” ([0068]). RUPPERSBERG further teaches analyzing the eardrum for “diagnosing an ear disease.” ([0133]). “[O]bjects shown in the at least on captured image may be identified (and distinguished from other objects in the subject's ear), and then the status (especially the temperature) of at least one of the identified objects is determined.” ([0133]). As previously discussed, this includes using infrared images that show a temperature distribution of the tympanic membrane. (see, e.g., [0051]-[0052])).
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to be configured to identify infection areas within the ear canal based at least in part on the compensated subset of the sensor data and the coverage metric. One of ordinary skill in the art would have desired to analyze the useful information of infrared images (i.e., the regions corresponding to the tympanic membrane and not obstructions) to determine signs of infection. DOUGLAS and RUPPERSBERG both teach it is possible to detect signs of infection using image data. RUPPERSBERG specifically teaches that infrared images may be helpful in diagnosing certain conditions.
With respect to claim 16, RUPPERSBERG, as modified by DOUGLAS and TSUBOI, teaches a method that includes determining a degree of variance as between the coverage metric and the coverage criterion, and wherein identifying the infection areas is based at least in part on the degree of variance.
RUPPERSBERG teaches that known device may incorrectly measure the temperature of the eardrum because the device may face a wall of the ear canal instead of the eardrum. ([0003]). “The reason for this is that a free line of sight from the infrared sensor unit to the eardrum or tympanic membrane is mandatory for correctly measuring the subject's body core temperature.” ([0003]).
To this end, RUPPERSBERG teaches using a wide angle video camera. “Such wide angle cameras allow detection of the subject's eardrum, even if the optical axis (‘main viewing direction’) of the camera is initially not directly centered to the eardrum. Once the eardrum has been detected in some region of the captured wide angle image, the operator of the ear inspection device may be informed, e.g. by some kind of guidance system, how to manipulate the position or orientation of the device with respect to the subject's ear so as to center the optical axis of the camera (and thus of the infrared sensor unit) to the eardrum.” Paragraph [0212] describes guiding the user in positioning the otoscope properly, which may include indicating an insertion depth, a direction of rotation, a tilting angle. In order to guide a user, the system must be comparing the current position to another more optimal position.
It would have been obvious to one having ordinary skill in the art at the time of filing to modify the RUPPERSBERG device to determine a degree of variance as between the coverage metric and the coverage criterion, and wherein identifying the infection areas is based at least in part on the degree of variance. One would have been motivated to determine the degree of variance in order to guide the user into acquiring better data of the tympanic membrane. By acquiring better data, identification of the infection areas would necessarily be based on the degree of variance. As discussed above, the TSUBOI method (as modified by DOUGLAS) would analyze the images to determine how a user should move the device to acquire a better image. There would have been a reasonable expectation of success as RUPPERSBERG teaches that one can determine how to guide a user to a better position.
With respect to claim 17, RUPPERSBERG, as modified by DOUGLAS and TSUBOI, teaches a method that includes determining, from the sensor data, the occlusion area within the ear canal and determining that the occlusion area has impacted the coverage metric or an ability to detect the signs of the infection and outputting an alert based at least in part on determination that the occlusion area has impacted the coverage metric or the ability to detect the signs of the infection. As discussed above, the TSUBOI method (as modified by DOUGLAS) would identify the occlusion areas (e.g., obstructions). Each of RUPPERSBERG and DOUGLAS describe instructing the user to move the device if the amount of tympanic membrane that is viewable is not sufficient. TSUBOI uses a classifier to determine an “occupancy rate” that is representation of how much of the tympanic membrane is viewable. As such, one having ordinary skill in the art would have been motivated to modify the TSUBOI classifier to not only determine which pixels correspond to the tympanic membrane but also to determine whether the coverage metric satisfies a coverage criterion for detecting infection areas. If the coverage metric does not satisfy the criterion (e.g., at least 60% of the tympanic membrane), the user could be warned or guided in moving the device, as taught in RUPPERSBERG and DOUGLAS.
With respect to claim 19, RUPPERSBERG does not explicitly teach monitoring, over a period of time, temperature readings from the sensor array; determining, from the temperature readings received over the period of time, that the infection has worsened; and providing a notification that the infection has worsened.
DOUGLAS teaches various embodiments for acquiring images of the tympanic membrane. ([0014]). DOUGLAS’s embodiments may configure “for use with a home or clinical device that includes an otoscope (e.g., speculum, lens/lenses, and video/image capture capability). Images may be acquired until the method/apparatus indicates, e.g., visually or audibly, that an adequate image has been taken. The image(s) may then be stored, transmitted, and/or analyzed. For example, stored images may be transmitted to a medical provider for further analysis, or to a third-party analysis center.” ([0034]).
DOUGLAS also teaches tracking treatment progress. “Once a diagnosis is made and an intervention is prescribed, it is helpful to track the progression of the disease via continuous exams and to suggest altering interventions if the disease is progressing or otherwise not improving as rapidly as expected. Methods and software tools can be a powerful aid in this tracking.” ([0265]). “A method for continuous monitoring is shown in FIG. 20. Initially, an exam is performed, a diagnosis is made (possibly with help from the automated diagnosis method, described herein) and an intervention (including, possibly, no intervention or “watchful waiting”) is prescribed (possibly with help from the automated prognosis method). After waiting for some period, e.g., one day, the exam is repeated. If the disease is improving as expected (possibly as determined with help from the automated method to determine disease stage) but has not yet resolved, the waiting period plus exam cycle may be repeated. If the disease is not improving as expected, a new intervention is prescribed based on the results of the most recent exam and, possibly, prior exams (again, possibly with help from the automated prognosis method).” ([0266]). By recommending a new intervention using the automated prognosis method, the user would be notified that the infection has worsened.
It would have been obvious to one having ordinary skill in the art at the time of filing to monitor, over a period of time, temperature readings from the sensor array, determine that the infection has worsened, and provide a notification that the infection has worsened, as taught in DOUGLAS. One of ordinary skill in the art would have been motivated to configured the RUPPERSBERG device to monitor and notify the user about the progression of the infection so that the user would know if the intervention is working. There would have been a reasonable expectation of success as DOUGLAS teaches that one can monitor the progression of an ear infection and keep the user updated on the progress.
With respect to claim 20, RUPPERSBERG teaches determining heat distribution asymmetries as between first temperature readings of a first ear of the patient and second temperature readings of a second ear of the patient, and wherein detecting the infection is based at least in part on comparative diagnosis performed on the first temperature readings and the second temperature readings. More specifically, RUPPERSBERG teaches that “[i]f an elevated temperature (i.e. a temperature above the normal body core temperature of a human being) is detected by the infrared sensor unit when the ear inspection device according to the present invention is introduced at least partially in one of the two exterior ear canals of the subject, this does not always allow to conclude that the subject has an elevated body core temperature, i.e. fever. Instead, the measured elevated temperature may result from a local inflammation of the eardrum of the ear into which the device has been introduced. Local inflammations also lead to a raise in temperature at the site of inflammation. To distinguish between these two cases, i.e. fever vs. local inflammation, it is advantageous to subsequently carry out the temperature measurement at both (i.e. left and right) ears of the subject.” ([0028]).
Claims 4, 7, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Appl. Publ. No. 2015/0351637 A1 (hereinafter “RUPPERSBERG”) and U.S. Patent Appl. Publ. No. 2013/0296685 A1 (hereinafter “TSUBOI”) and U.S. Patent Appl. Publ. No. 2015/0065803 A1 (hereinafter “DOUGLAS”) as applied to claims 1 and 15 above, and further in view of U.S. Patent No. 10,764,514 (hereinafter “HOEVENAAR”).
With respect to claim 4, RUPPERSBERG teaches calibrating for detection precision (see, e.g., [0208]) but RUPPERSBERG does not explicitly teach adjusting a sensitivity of the sensor.
HOEVENAAR teaches systems and computer-implemented methods for controlling a gain state of a thermal camera having an infrared sensor. (Abstract and col. 10, lines 60-63). HOEVENAAR observes that thermal cameras are configured to operate in different gain states. “[A] thermal camera can be configured to operate in either a high-gain state or a low-gain state. The high-gain state may provide a first, high accuracy and support a narrow intra-scene range of temperatures, whereas, the low-gain state may provide a second, lower accuracy but support a higher intra-scene range of temperatures.” (Col. 1, lines 17-22).
“The gain state can control a temperature sensitivity of the camera module or an intra-scene range of the thermal camera.” (col. 9, lines 51-53). Notably, HOEVENAAR describes sensor arrays similar to the ones described in HARR. “Infrared sensor 56 can include an array of thermal detectors and a readout component. As an example, infrared sensor can include an 80×60 array of thermal detectors that is configured to detect infrared radiation in a response wavelength band. Each thermal detector may have an element, such as a microbolometer, whose temperature fluctuates in response to incident flux…As another example, each thermal detector can include a thermopile sensor….” (Col. 10, line 66 to col. 11, line 8).
HOEVENAAR describes adjusting gain states if the sensors are saturated. “When the thermal camera operates in the high-gain state and a scene imaged by the thermal camera includes temperatures above 140° C., those temperatures may saturate sensors of the thermal camera. To view the temperatures above 140° C., the thermal camera can be switched to the low-gain state, so that the temperatures above 140° C. may be more accurately quantified and visualized.” Sensors are saturated when the intensity of IR radiation hitting the sensor is too high.
It would have been obvious to one having ordinary skill in the art to identify adjust a sensitivity of the sensor. If the IR sensor is saturated (i.e., the infrared radiation variation is too high), one skilled in the art would be motivated to configure the system to switch to a low gain state, based on the teachings of HOEVENAAR, and then acquire the temperature information so that a more accurate measurement of the temperature information is acquired. There would be a reasonable expectation of success as HOEVENAAR teaches that thermopile sensors can be configured to adjust gain states if the sensors are saturated.
With respect to claim 7, RUPPERSBERG does not teach determining a heat value from the sensor data; determining an infrared value from the sensor data; and determining a severity of the infection based at least in part on the heat value and the infrared value.
HOEVENAAR describes adjusting gain states if the sensors are saturated. Sensors are saturated when the intensity of IR radiation (i.e., the infrared value) is too high. “When the thermal camera operates in the high-gain state and a scene imaged by the thermal camera includes temperatures above 140° C., those temperatures may saturate sensors of the thermal camera. To view the temperatures above 140° C., the thermal camera can be switched to the low-gain state, so that the temperatures above 140° C. may be more accurately quantified and visualized.” The heat value is based on the infrared value, which causes a change in gain state if the infrared value is too high. As such, the severity of the infection would be based, at least in part, on the heat value and the infrared value.
It would have been obvious to one having ordinary skill in the art to consider the infrared radiation values when evaluating the temperature measurements. If the IR sensor is saturated (i.e., the infrared radiation variation is too high), one skilled in the art would configure the system to switch to a low gain state, based on the teachings of HOEVENAAR, and then acquire the heat values. Any detection of acute otitis media would be based, at least in part, on detecting the infrared radiation values and detecting the heat values.
With respect to claim 18, RUPPERSBERG does not teach adjusting a sensitivity of the sensor array based at least in part on the coverage metric and whether the occlusion area impacts the coverage metric, wherein adjusting the sensitivity includes weighting or excluding sensor data associated with grid-coordinate locations that fail to satisfy the coverage criterion or are associated with the occlusion area.
HOEVENAAR teaches systems and computer-implemented methods for controlling a gain state of a thermal camera having an infrared sensor. (Abstract and col. 10, lines 60-63). HOEVENAAR observes that thermal cameras are configured to operate in different gain states. “[A] thermal camera can be configured to operate in either a high-gain state or a low-gain state. The high-gain state may provide a first, high accuracy and support a narrow intra-scene range of temperatures, whereas, the low-gain state may provide a second, lower accuracy but support a higher intra-scene range of temperatures.” (Col. 1, lines 17-22).
“The gain state can control a temperature sensitivity of the camera module or an intra-scene range of the thermal camera.” (col. 9, lines 51-53). Notably, HOEVENAAR describes sensor arrays similar to the ones described in HARR. “Infrared sensor 56 can include an array of thermal detectors and a readout component. As an example, infrared sensor can include an 80×60 array of thermal detectors that is configured to detect infrared radiation in a response wavelength band. Each thermal detector may have an element, such as a microbolometer, whose temperature fluctuates in response to incident flux…As another example, each thermal detector can include a thermopile sensor….” (Col. 10, line 66 to col. 11, line 8).
HOEVENAAR describes adjusting gain states if the sensors are saturated. “When the thermal camera operates in the high-gain state and a scene imaged by the thermal camera includes temperatures above 140° C., those temperatures may saturate sensors of the thermal camera. To view the temperatures above 140° C., the thermal camera can be switched to the low-gain state, so that the temperatures above 140° C. may be more accurately quantified and visualized.” Sensors are saturated when the intensity of IR radiation hitting the sensor is too high.
It would have been obvious to one having ordinary skill in the art to identify adjust a sensitivity of the sensor. If the IR sensor is saturated (i.e., the infrared radiation variation is too high), one skilled in the art would be motivated to configure the system to switch to a low gain state, based on the teachings of HOEVENAAR, and then acquire the temperature information so that a more accurate measurement of the temperature information is acquired. An amount of infrared radiation that is detected by the sensor array is based, at least in part, on the position of the device. As such, the sensitivity adjustment would be based, at least in part, on the position of the device. There would be a reasonable expectation of success as HOEVENAAR teaches that thermopile sensors can be configured to adjust gain states if the sensors are saturated.
RESPONSE TO APPLICANT’S ARGUMENTS
Applicant’s arguments with respect to claims 1-20 have been considered but are moot because the new grounds of rejection do not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Prior Art of Record
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US- 20190216308-A1 teaches methods of automatically classifying pathologies of the tympanic membrane using a classification model.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON P GROSS whose telephone number is (571)272-1386. The examiner can normally be reached Monday-Friday 9:00-5:00CT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne M. Kozak can be reached at (571) 270-5284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON P GROSS/Examiner, Art Unit 3797
/SERKAN AKAR/Primary Examiner, Art Unit 3797
Prosecution Insights