Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-12, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Giovinazzo et al. (US 20190191995 A1) in view of Solanki et al. (US 20200257879 A1) and further in view of Birkne et al. (US 20210045630 A1).
Regarding claim 1, Giovinazzo et al. teaches a diagnostic image management system for managing diagnostic images used as an input data for a diagnostic model to generate diagnostic assistance information for eyes as a diagnostic target comprising (see para [0005]; “systems and methods for determining a level of one or more chemical substances in a subject, including the steps of: (a) capturing a plurality of images of one or both of a subject's pupils on a device capable of capturing multiple images over several seconds; (b) extracting pupil measurements as a function of time from the plurality of images so as to determine one or more PLRs; and (c) analyzing the PLR so as to provide a diagnostic output”): a user terminal configured to allow taking a captured image only when pre-stored capturing conditions are satisfied (see Fig. 5, para [0065]; “Only after various conditions are within tolerance ranges will the software automatically begin a countdown to recording. Conditions must remain in range throughout the recording”, see also para [0080]; “A countdown phase 30 automatically begins when the image is within tolerances….The system monitors for changes in positioning/lighting/distance/occlusion/etc. and stops scanning if a parameter falls outside predefined tolerances”), obtain a position information regarding the diagnostic target using a first algorithm, and transmit the position information and the captured image to a server (see para [0063]; “(d) Boundary drawings (image segmentation) in real-time that “find” the general location of the iris and the pupil such that these x,y coordinates (and in some case full mappings) are reported back to the server with each frame”, see para [0089]; “The data package sent to the server at the end of the scan/capture includes….. ambient light level, distance estimate, remotely-configured settings (FPS rate, minimum ambient light, dark iris threshold, etc.); and [0093] 4) Per frame data: ambient light level, distance estimate, location point data for each eye”, and para [0055]; “In Step 2, the video is transmitted to a server for processing”), wherein the server is configured to: receive the captured image and the position information from the user terminal (see para [0055]; “In Step 1, a video of the PLR is captured by a medical personnel or first responder, preferably on a personal electronic device. In Step 2, the video is transmitted to a server for processing” see also para [0059]; “Pupil diameter measurements are made at the handheld device and transferred to the back-end server” see para [0089]; “The data package sent to the server at the end of the scan/capture includes….. 4) Per frame data: ambient light level, distance estimate, location point data for each eye”), determine, for the captured image, verified information regarding the diagnostic target using a second algorithm on the captured image (see para [0030]; “processing image data from the one or more videos so as to extract pupil measurements as a function of time from the image data; and (d) determining one or more PLRs based on the pupil measurements”, see also para [0036]; “extracting pupil measurements as a function of time from the plurality of images so as to determine one or more PLRs; and (c) analyzing the PLR so as to provide a diagnostic output that identifies”, Note: this is a secondary algorithmic evaluation on the server side computing measurements and analyzing them against prior models). However, Giovinazzo et al. does not teach the server configured to store a diagnostic image based on the captured image, store the captured image as the diagnostic image in response to determining that the captured image satisfies pre-stored verification conditions, and transmit a re-capture request to the user terminal in response to determining that the captured image does not satisfy the pre-stored verification conditions, wherein a first algorithm used is configured to require a higher computational load than an algorithm, compare the verified information and the position information, and wherein the pre-stored verification conditions comprise whether at least a difference between the position information and the verified information is less than a predetermined threshold or not.
In the same field of endeavor Solanki et al. disclose and the server configured to store a diagnostic image based on the captured image (see para [0009]; “a patient's retina imaging data and/or eye tracking data may be captured, analyzed and/or stored in one or more sessions and retrieved by the system for use during a diagnostic procedure”), store the captured image as the diagnostic image in response to determining that the captured image satisfies pre-stored verification conditions (see para [0448]; “The cloud infrastructure generates patient-level diagnostic reports which can trickle back to the patients, for example, through the same pipeline, in reverse” Note: post-verification workflow), and transmit a re-capture request to the user terminal in response to determining that the captured image does not satisfy the pre-stored verification conditions (see para [0247]; “Good quality images 179010 are sent for further processing for example on the cloud 179014, a computer or computing device 179004, a mobile device 179008, or the like. Poor quality images are rejected, and the operator is asked to retake the image”, see also para [0449]; “enabling the clinicians or ophthalmologists to discuss the results with the patients during their imaging visit. It also enables seamless re-imaging in cases where conclusive results could not be obtained using the initial images”), wherein a first algorithm is configured to require a higher computational load than an algorithm (see para [0445]; “Large-scale retinal image processing and analysis may not be feasible on normal desktop computers or mobile devices”, see also para [0457]; “each worker machine 459414 can comprise of 32 or more, multi-core, 64-bit processors with high computing power” Note: implies heavier compute in the cloud vs device, may not feasible on normal desktop computers or mobile device). Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date to the claimed invention to modify a method for capturing a pupillary light reflex (PLR) by capturing images of a subject's pupil using a smartphone, extracting image data to determine PLR and classifying the PLR to provide an analytical output of Giovinazzo et al. in view of a method that aid in screening, diagnosis and/or monitoring of medical conditions of Solanki et al. in order to improve first pass gradability, reduce operator burden and place heavy ML server side with lightweight capture checks device side (see para [0450]). However, the combination of Giovinazzo et al. and Solanki et al. as a whole does not teach compare the verified information and the position information, and wherein the pre-stored verification conditions comprise whether at least a difference between the position information and the verified information is less than a predetermined threshold or not.
In the same field of endeavor, Birkner et al. teach compare the verified information and the position information, and wherein the pre-stored verification conditions comprise whether at least a difference between the position information and the verified information is less than a predetermined threshold or not (see para [0007]; “The eye fixation parameters may comprise a reference position and orientation of the eye when fixated, …..determine a current eye position and orientation relative to the reference position and orientation of the eye when fixated. The control processor may be further configured to track the eye position and orientation and calculate an offset from the reference position and orientation. In some embodiments, when the offset is less than a threshold value, the eye may be determined to be fixated and the control processor generates an indication of fixation. If the offset is greater than the threshold value, the eye is determined to be out of alignment and the control processor generates an indication that the eye is not properly aligned”, see also para [0016]; “track the eye position and orientation and calculate an offset from the eye fixation parameters and determine if the offset is less than a threshold value. When the offset is less than the threshold value the eye is determined to be fixated and the control processor generates an indication of fixation. When the offset is greater than the threshold value the eye is determined to be out of alignment” Note: the reference explicitly compares them via an offset and tests against a predetermined threshold i.e., pass if less threshold, fail if greater or equal threshold). Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filling date to the claimed invention to modify a method for capturing a pupillary light reflex (PLR) by capturing images of a subject's pupil using a smartphone, extracting image data to determine PLR and classifying the PLR to provide an analytical output of Giovinazzo et al. in view of a method that aid in screening, diagnosis and/or monitoring of medical conditions of Solanki et al. and a method for tracking eye movement during a diagnostic procedure include a retina imaging system of Birkner et al. in order to improve tracking the position and orientation of a patient's eye during an ophthalmic procedure (see para [0450]).
Regarding claim 2, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein a first capturing parameter includes information on a detected position of a region including the diagnosis target in a pre-view image generated by the user terminal (see para [0060]; “(a) The location of the center of the pupil in each frame”, see also para [0063]; “(d) Boundary drawings (image segmentation) in real-time that “find” the general location of the iris and the pupil such that these x,y coordinates (and in some case full mappings) are reported back to the server with each frame of video”), and the user terminal is configured to compare the first capturing parameter with a pre-stored first area (see para [0065]; “Only after various conditions are within tolerance ranges will the software automatically begin a countdown to recording. Conditions must remain in range throughout the recording period; if they go out of range, the recording is stopped and the user informed”).
Regarding claim 3, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein a second capturing parameter includes information on an angle of a region including the diagnostic target in a pre-view image (see para [0362]; “combining image-level descriptors to form either the image field-of-view (identified from meta data or by using position of optic nerve head and macula)-specific”, see also para [0373]; “combining image-level descriptors to form either the image field-of-view, specific descriptors (identified from metadata or by using position of ONH as described in the section above entitled “Optic Nerve Head Detection” or by using the position of the ONH and macula) or eye-specific descriptors (identified from metadata or position of ONH and macula or the vector from the focus to the vertex of the parabola that approximates the major vascular arch)” Note: extracting an orientation/angle descriptor for the retina region that contains the diagnostic target), and the user terminal is configured to compare the second capturing parameter with a pre-stored angle of the region including the diagnostic target (see para [0022]; “These images may be registered using the disclosed system. This registration allows for the alignment of images such that the anatomical structures overlap”, see also para [0194]; “Image-to-image registration includes automated alignment of various structures of an image with another image of the same object possibly taken at a different time or different angle, different zoom, or a different field of imaging, where different regions are imaged with a small overlap”).
Regarding claim 4, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein a first verification parameter for an object used to determine a satisfaction of at least one of the pre-stored verification conditions (see para [0080]; “The system monitors for changes in positioning/lighting/distance/occlusion/etc. and stops scanning if a parameter falls outside predefined tolerances. When the scan is completed, the data is verified.. The verified data is then compiled and uploaded to the server”) includes information on a detected position of a region including the diagnosis target in the captured image (see Fig. 1, para [0063]; Boundary drawings (image segmentation) in real-time that “find” the general location of the iris and the pupil such that these x,y coordinates (and in some case full mappings) are reported back to the server with each frame of video” see also para [0059]; “The location of the center of the pupil in each frame”), and the server is configured to by compare the first verification parameter with a pre-stored third area (see para [0055]; “In Step 5, the results are returned to the medical personnel of first responder in either lexical format or assay format. Optionally, following Step 5, the recipient may provide feedback as to the accuracy of the result to facilitate improved accuracy by way of machine learning”, see also para [0100]).
Regarding claim 5, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein the algorithm used to determine the satisfaction of the pre-stored capturing conditions comprises a first landmark detection algorithm (see para [0065]- [0066]; “Only after various conditions are within tolerance ranges will the software automatically begin a countdown to recording. Conditions must remain in range throughout the recording period; if they go out of range, the recording is stopped and the user informed….One feature is an open source eye-tracking software package such as Drishti (Drishti Technologies, Inc., Palo Alto, Calif.). Drishti provides, for each eye in each frame of a video, 27 eye positioning points. Running Drishti, or similar software, on the client side provides real-time positional guidance to the user before and during the recording process”), wherein the first capturing parameter is a first landmark of the eye and includes information on a detected position of the diagnostic target in the pre-view image (see para [0063]; “(d) Boundary drawings (image segmentation) in real-time that “find” the general location of the iris and the pupil such that these x,y coordinates (and in some case full mappings) are reported back to the server with each frame of video”, see also para [0080]; “A countdown phase 30 automatically begins when the image is within tolerances.. The system monitors for changes in positioning/lighting/distance/occlusion/etc. and stops scanning if a parameter falls outside predefined tolerances” Note: It compares the measured parameters to preset ranges before starting the scan).
Solanki et al. in the combination further teach wherein the first algorithm used to determine a satisfaction of at least one of the pre- stored verification conditions is a second landmark detection algorithm, and wherein the first verification parameter is a second landmark of the eye and includes information on a detected position of the diagnostic target in the captured image (see para [0139]; “it may be beneficial to detect the optic nerve heard (ONH) within a retinal image. A ONH can be robustly detected using an approach that mirrors the one for lesions as described in section below entitled “Lesion Localization”. In another embodiment, multi-resolution decomposition and template matching is employed for ONH localization”, see also para [0142]; “these approximate locations and sizes for the ONH can be refined by performing template matching in a neighborhood about these approximate ONH locations and choosing the one location and size that gives the maximum confidence or probability of ONH presence”).
Regarding claim 6, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein the algorithm used to determine the satisfaction of the pre-stored capturing conditions comprises a first landmark detection algorithm (see para [0065]- [0066]; “Only after various conditions are within tolerance ranges will the software automatically begin a countdown to recording. Conditions must remain in range throughout the recording period; if they go out of range, the recording is stopped and the user informed….One feature is an open source eye-tracking software package such as Drishti (Drishti Technologies, Inc., Palo Alto, Calif.). Drishti provides, for each eye in each frame of a video, 27 eye positioning points. Running Drishti, or similar software, on the client side provides real-time positional guidance to the user before and during the recording process”), wherein the first capturing parameter for an object used to determine a satisfaction of at least one of the pre-stored verification conditions is a first landmark of the eye and includes information on a detected position of the diagnostic target in the pre-view image (see para [0063]; “(d) Boundary drawings (image segmentation) in real-time that “find” the general location of the iris and the pupil such that these x,y coordinates (and in some case full mappings) are reported back to the server with each frame of video”, see also para [0080]; “A countdown phase 30 automatically begins when the image is within tolerances.. The system monitors for changes in positioning/lighting/distance/occlusion/etc. and stops scanning if a parameter falls outside predefined tolerances” Note: It compares the measured parameters to preset ranges before starting the scan).
Solanki et al. in the combination further teach wherein the first algorithm used to determine a satisfaction of at least one of the pre-stored verification conditions is an image segmentation algorithm (see para [0148]; “These images will have certain similar characteristics that can be utilized for various tasks, such as image segmentation, detection, or analysis”), and wherein the first verification parameter is an area of the eye and includes information on a detected position of the diagnostic target in the captured image (see para [0139]; “it may be beneficial to detect the optic nerve heard (ONH) within a retinal image. A ONH can be robustly detected using an approach that mirrors the one for lesions as described in section below entitled “Lesion Localization”. In another embodiment, multi-resolution decomposition and template matching is employed for ONH localization”, see also para [0142]; “these approximate locations and sizes for the ONH can be refined by performing template matching in a neighborhood about these approximate ONH locations and choosing the one location and size that gives the maximum confidence or probability of ONH presence”).
Regarding claim 7, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein the user terminal is configured to transmit a first capturing parameter to the server in response to determining that the pre-stored capturing conditions are satisfied (see para [0080]; “The verified data is then compiled and uploaded to the server”), and wherein the pre-stored verification conditions comprises a first verification condition related to a first verification parameter and the first capturing parameter (see para [0059]; “High quality scans (videos) for analysis by the back-end server and a process to obtain those videos with metadata attached is described with reference to FIG. 5. Including metadata enables the back-end server to do a better job processing images. Pupil diameter measurements are made at the handheld device and transferred to the back-end server as metadata appended to the frame. This embodiment enables relatively advanced image analysis in real time at the handheld device by guiding a clinician to take an optimal scan and then provides metadata so that the back-end processing can more easily locate and measure the pupil, the iris, and other features/movements of the eye. When the system is running, it is identifying”).
Regarding claim 8, the rejection of claim 7 is incorporated herein.
Giovinazzo et al. et al. in the combination further teach wherein the server is configured to determine whether the first verification condition is satisfied by comparing a difference between a first verification parameter and the first capturing parameter with a threshold value (see para [0059]; “High quality scans (videos) for analysis by the back-end server and a process to obtain those videos with metadata attached is described with reference to FIG. 5. Including metadata enables the back-end server to do a better job processing images. Pupil diameter measurements are made at the handheld device and transferred to the back-end server as metadata appended to the frame. This embodiment enables relatively advanced image analysis in real time at the handheld device by guiding a clinician to take an optimal scan and then provides metadata so that the back-end processing can more easily locate and measure the pupil, the iris, and other features/movements of the eye. When the system is running, it is identifying”, see also para [0061]- [0065] transmits the captured parameter (e.g., pupil x,y; pupil diameter) to the server with each frame, so the server has the capture time value. Server verification ad threshold performs server-side localization/quality and already uses explicit threshold (focus, exposure) and difference operations).
Regarding claim 9, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein the user terminal is configured to obtain a pre-view images according to a preset frame rate (see para [0067]; “One example would be the frames per second (FPS) rate of the video during capture”), analyze at least a part of the pre-view images with second algorithm (see para [0095]; “Step 4: FIG. 7 schematically illustrates a neural network 38 where the 270 measurements are used as inputs into 270 input nodes 40 of a multi-layer “deep learning” back-propagation neural network that has output nodes 42 corresponding to specific substances and substance types, and hidden layers suitable to support a convolutional neural network”, see also para [0096]; “Many other proprietary and non-proprietary classification tools are used for Step 4, always competing for accuracy with the multi-layer deep-learning back-propagation neural network. These methods include, but are not limited to, Support Vector Machines, Graph-Theory-Based Classification Algorithms, and Feed-Forward (unsupervised) learning networks”), and transmit one of the pre-view images as the captured image to the server (see para [0055]; “In Step 1, a video of the PLR is captured by a medical personnel or first responder, preferably on a personal electronic device. In Step 2, the video is transmitted to a server for processing”).
Regarding claim 10, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein the user terminal is configured to, in response to determining that a first capturing condition for a preview image is not satisfied, analyze a second pre-view image with the algorithm used to determine the satisfaction of pre-stored capturing condition, wherein the second pre-view image is obtained after it is determined that the first capturing condition for the pre-view image is not satisfied (see para [0063]; “(d) Boundary drawings (image segmentation) in real-time that “find” the general location of the iris and the pupil such that these x,y coordinates (and in some case full mappings) are reported back to the server with each frame of video”, see also para [0065]; “Only after various conditions are within tolerance ranges will the software automatically begin a countdown to recording. Conditions must remain in range throughout the recording period; if they go out of range, the recording is stopped and the user informed”, and para [0033]; “using parameters that are adjusted manually or automatically. In one embodiment, the adjusting is carried out based on real-time video capture results” Note: when a preview frame fails tolerance, it continues analyzing subsequent preview frames with the same landmark/segmentation algorithm until conditions are met- i.e., the “second preview image” obtained after the failure is analyzed by the same algorithm).
Regarding claim 11, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. further teach wherein the user terminal is configured to, in response to determining that a first capturing condition for the pre-view image is satisfied, transmit a second pre-view image as the captured image to the server, and wherein the second pre-view image is captured after it is determined that the first capturing condition for a first image is satisfied (see para [0063]; “(d) Boundary drawings (image segmentation) in real-time that “find” the general location of the iris and the pupil such that these x,y coordinates (and in some case full mappings) are reported back to the server with each frame of video”, see also para [0065]; “Only after various conditions are within tolerance ranges will the software automatically begin a countdown to recording. Conditions must remain in range throughout the recording period; if they go out of range, the recording is stopped and the user informed”, and para [0033]; “using parameters that are adjusted manually or automatically. In one embodiment, the adjusting is carried out based on real-time video capture results”).
Regarding claim 12, the rejection of claim 1 is incorporated herein.
Giovinazzo et al. in the combination further teach wherein in the user terminal, a period of obtaining a first capturing parameter and a period of obtaining the second capturing parameter are different from each other (see para [0031]; “parameters for the flash of light are pre-set in the PED or adjusted manually or adjusted automatically. In one embodiment, the parameters are selected from the group consisting of wavelength, pattern, duration, frequency and distance from eye. In one embodiment, the spectrum of the wavelength of the flash of light is in the visible light spectrum (nominally from 400 nanometers to 700 nanometers). In another embodiment, the spectrum of the wavelength of the flash of light is in the infrared spectrum (nominally from 700 nanometers to 1 millimeter). In one embodiment, the spectrum of the wavelength of the flash of light is about 450 nanometers. ..In one embodiment, the multiple flashes are continuous, random, or repeating as to flash duration and duration of time between flash illuminations”).
Regarding claim 21, the rejection of claim 1 is incorporated herein.
Solanki et al. in the combination further teach wherein the capturing parameters comprise at least one of information related to a landmark of a body part including the diagnostic target (see para [0022]; “prediction for each pixel for each class of lesion or retinal anatomical structure, such as optic nerve head, veins, arteries, and/or fovea”, see also para [0194]; “When applied to retinal images, registration can include identification of different structures in the retinal images that can be used as landmarks”), a position of a body part including the diagnostic target (see para [0140]; “the vertical position of the ONH is approximated by examining vertical image strips centered about the N approximate horizontal positions. This ONH position approximation technique produces M approximate locations for the ONH”), and a bounding box of a body part including the diagnostic target (see para [0311]; “Table 4 is one embodiment of blob properties used as descriptors…. Extent Ratio of pixels in the blob region to pixels in the total bounding box for the blob”).
Regarding claim 22, the rejection of claim 21 is incorporated herein.
Solanki et al. in the combination further teach wherein the verification parameters comprise at least one of information related to a landmark of a body part including the diagnostic target (see para [0022]; “prediction for each pixel for each class of lesion or retinal anatomical structure, such as optic nerve head, veins, arteries, and/or fovea”, see also para [0194]; “When applied to retinal images, registration can include identification of different structures in the retinal images that can be used as landmarks”),, a position of a body part including the diagnostic target (see para [0140]; “the vertical position of the ONH is approximated by examining vertical image strips centered about the N approximate horizontal positions. This ONH position approximation technique produces M approximate locations for the ONH”), and a bounding box of a body part including the diagnostic target (see para [0311]; “Table 4 is one embodiment of blob properties used as descriptors…. Extent Ratio of pixels in the blob region to pixels in the total bounding box for the blob”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WINTA GEBRESLASSIE whose telephone number is (571)272-3475. The examiner can normally be reached Monday-Friday9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WINTA GEBRESLASSIE/Examiner, Art Unit 2677