DETAILED ACTION
Contents
Notice of Pre-AIA or AIA Status 2
Claim Rejections - 35 USC § 103 2
Allowable Subject Matter 25
Conclusion 26
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to applicant’s claim set received on 9/16/24. Claims 1-22 are currently pending.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimedinvention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 12, 13, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Kalra et al (US 2021/0264607 A1) in view of Zhou et al (US 2004/0061834 A1).
Regarding claim 1, Kalra teaches a nerve detection system comprising: a general-purpose graphics processing unit (GPGPU) (see 0066; a graphics processing unit (GPU)) configured to receive raw polarized data and raw RGB image data (see 0063, 0065, 0066, 0109, 0114, 0153; As a result, the polarization camera captures multiple input images 18 (or polarization raw frames) of the scene 1, where each of the polarization raw frames 18 corresponds to an image taken behind a polarization filter or polarizer at a different angle of polarization ϕ.sub.pol (e.g., 0 degrees, 45 degrees, 90 degrees, or 135 degrees). Each of the polarization raw frames is captured from substantially the same pose with respect to the scene 1 (e.g., the images captured with the polarization filter at 0 degrees, 45 degrees, 90 degrees, or 135 degrees are all captured by a same polarization camera located at a same location and orientation), as opposed to capturing the polarization raw frames from disparate locations and orientations with respect to the scene. The polarization camera 10 may be configured to detect light in a variety of different portions of the electromagnetic spectrum, such as the human-visible portion of the electromagnetic spectrum, red, green, and blue portions of the human-visible spectrum, as well as invisible portions of the electromagnetic spectrum such as infrared and ultraviolet… The polarization raw frames 18 are supplied to a processing circuit 100, described in more detail below, computes a segmentation map 20 based of the polarization raw frames 18. As shown in FIG. 1, in the segmentation map 20, the transparent objects 2 and the opaque objects 3 of the scene are all individually labeled, where the labels are depicted in FIG. 1 using different colors or patterns (e.g., vertical lines, horizontal lines, checker patterns, etc.), but where, in practice, each label may be represented by a different value (e.g., an integer value, where the different patterns shown in the figures correspond to different values) in the segmentation map….. Accordingly, extracting features such as polarization feature maps or polarization images from polarization raw frames 18 produces first tensors 50 from which transparent objects or other optically challenging objects such as translucent objects, multipath inducing objects, non-Lambertian objects, and non-reflective objects are more easily detected or separated from other objects in a scene. In some embodiments, the first tensors extracted by the feature extractor 800 may be explicitly derived features (e.g., hand crafted by a human designer) that relate to underlying physical phenomena that may be exhibited in the polarization raw frames (e.g., the calculation of AOLP and DOLP images, as discussed above). In some additional embodiments of the present disclosure, the feature extractor 800 extracts other non-polarization feature maps or non-polarization images, such as intensity maps for different colors of light (e.g., red, green, and blue light) and transformations of the intensity maps (e.g., applying image processing filters to the intensity maps). In some embodiments of the present disclosure the feature extractor 800 may be configured to extract one or more features that are automatically learned (e.g., features that are not manually specified by a human) through an end-to-end supervised training process based on labeled training data…. In some embodiments of the present disclosure, a same predictor or statistical model 900 is trained to detect both transparent objects and opaque objects (or to generate second tensors C in second representation space) based on training data containing labeled examples of both transparent objects and opaque objects. For example, in some such embodiments, a Polarized CNN architecture is used, such as the Polarized Mask R-CNN architecture shown in FIG. 9. In some embodiments, the Polarized Mask R-CNN architecture shown in FIG. 9 is further modified by adding one or more additional CNN backbones that compute one or more additional mode tensors. The additional CNN backbones may be trained based on additional first tensors. In some embodiments these additional first tensors include image maps computed based on color intensity images (e.g., intensity of light in different wavelengths, such as a red intensity image or color channel, a green intensity image or color channel, and a blue intensity image or color channel). In some embodiments, these additional first tensors include image maps computed based on combinations of color intensity images. In some embodiments, the fusion modules 920 fuse all of the mode tensors at each scale from each of the CNN backbones (e.g., including the additional CNN backbones).), generate output based on the raw polarized data (see 0109; Accordingly, extracting features such as polarization feature maps or polarization images from polarization raw frames 18 produces first tensors 50 from which transparent objects or other optically challenging objects such as translucent objects, multipath inducing objects, non-Lambertian objects, and non-reflective objects are more easily detected or separated from other objects in a scene. In some embodiments, the first tensors extracted by the feature extractor 800 may be explicitly derived features (e.g., hand crafted by a human designer) that relate to underlying physical phenomena that may be exhibited in the polarization raw frames (e.g., the calculation of AOLP and DOLP images, as discussed above). In some additional embodiments of the present disclosure, the feature extractor 800 extracts other non-polarization feature maps or non-polarization images, such as intensity maps for different colors of light (e.g., red, green, and blue light) and transformations of the intensity maps (e.g., applying image processing filters to the intensity maps). In some embodiments of the present disclosure the feature extractor 800 may be configured to extract one or more features that are automatically learned (e.g., features that are not manually specified by a human) through an end-to-end supervised training process based on labeled training data.), and simultaneously process the raw RGB data (see 0125-0128, 0153; In the embodiment shown in FIG. 9, derived feature maps 50 (e.g., including input polarization images such as AOLP ϕ and DOLP ρ images) are supplied as inputs to a Polarized CNN backbone 910. In the embodiment shown in FIG. 9, the input feature maps 50 include three input images: the intensity image (I) 52, the AOLP (P) 56, the DOLP (p) 54 from equation (1) as the input for detecting a transparent object and/or other optically challenging object. These images are computed from polarization raw frames 18 (e.g., images I.sub.0, I.sub.45, I.sub.90, and I.sub.135 as described above), normalized to be in a range (e.g., 8-bit values in the range [0-255]) and transformed into three-channel gray scale images to allow for easy transfer learning based on networks pre-trained on the MSCoCo dataset (see, e.g., Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.). [0126] In the embodiment shown in FIG. 9, each of the input derived feature maps 50 is supplied to a separate backbone: intensity B.sub.I(I) 912, AOLP backbone B.sub.ϕ(ϕ) 914, and DOLP backbone B.sub.ρ(φ 916. The CNN backbones 912, 914, and 916 compute tensors for each mode, or “mode tensors” (e.g., feature maps computed based on parameters learned during training or transfer learning of the CNN backbone, discussed in more detail below) C.sub.i,I, C.sub.i,ρ, C.sub.i,ϕ at different scales or resolutions i. While FIG. 9 illustrates an embodiment with five different scales i, embodiments of the present disclosure are not limited thereto and may also be applied to CNN backbones with different numbers of scales. [0127] Some aspects of embodiments of the present disclosure relate to a spatially-aware attention-fusion mechanism to perform multi-modal fusion (e.g., fusion of the feature maps computed from each of the different modes or different types of input feature maps, such as the intensity feature map I, the AOLP feature map ϕ, and the DOLP feature map ρ). [0128] For example, in the embodiment shown in FIG. 9, the mode tensors C.sub.i,I, C.sub.i,ρ, C.sub.i,ϕ (tensors for each mode) computed from corresponding backbones B.sub.I, B.sub.ρ, B.sub.ϕ at each scale i are fused using fusion layers 922, 923, 924, 925 (collectively, fusion layers 920) for corresponding scales. For example, fusion layer 922 is configured to fuse mode tensors C.sub.2,I, C.sub.2,ρ, C.sub.2,ϕ computed at scale i=2 to compute a fused tensor C.sub.2. Likewise, fusion layer 923 is configured to fuse mode tensors C.sub.3,I, C.sub.3,ρ, C.sub.3,ϕ computed at scale i=3 to compute a fused tensor C.sub.3, and similar computations may be performed by fusion layers 924 and 925 to compute fused feature maps C.sub.4 and C.sub.5, respectively, based on respective mode tensors for their scales. The fused tensors C.sub.i (e.g., C.sub.2, C.sub.3, C.sub.4, C.sub.5), or second tensors, such as fused feature maps, computed by the fusion layers 920 are then supplied as input to a prediction module 950, which is configured to compute a prediction from the fused tensors, where the prediction may be an output such as a segmentation map 20, a classification, a textual description, or the like…. In some embodiments of the present disclosure, a same predictor or statistical model 900 is trained to detect both transparent objects and opaque objects (or to generate second tensors C in second representation space) based on training data containing labeled examples of both transparent objects and opaque objects. For example, in some such embodiments, a Polarized CNN architecture is used, such as the Polarized Mask R-CNN architecture shown in FIG. 9. In some embodiments, the Polarized Mask R-CNN architecture shown in FIG. 9 is further modified by adding one or more additional CNN backbones that compute one or more additional mode tensors. The additional CNN backbones may be trained based on additional first tensors. In some embodiments these additional first tensors include image maps computed based on color intensity images (e.g., intensity of light in different wavelengths, such as a red intensity image or color channel, a green intensity image or color channel, and a blue intensity image or color channel). In some embodiments, these additional first tensors include image maps computed based on combinations of color intensity images. In some embodiments, the fusion modules 920 fuse all of the mode tensors at each scale from each of the CNN backbones (e.g., including the additional CNN backbones). Kalra does not teach expressly a birefringence and the birefringence output to provide nerve detection information.
Zhou, in the same field of endeavor, teaches a birefringence and the birefringence output to provide nerve detection information (see 0027; 0045-0046; In one aspect, the invention is a method for analyzing the birefringence of a retinal structure of an eye including the steps of (a) producing a first polarimetric image of the retinal structure having a plurality of pixels {S(.delta..sub.T, .theta..sub.T)} each representing a measured retardance magnitude .delta..sub.T and orientation angle .theta..sub.T, (b) determining the fast axes of the pixel retardance angles {.theta..sub.T} corresponding to a biological feature of the retinal structure in the first polarimetric image, (c) determining anterior segment retardance magnitude .delta..sub.C and orientation .theta..sub.C values corresponding to the variation of the retardance magnitude .delta..sub.T over an annular region of the first polarimetric image, and (d) computing a second polarimetric image of the retinal structure having a plurality of pixels {S(.delta..sub.N, .theta..sub.N)} each corresponding to a first polarimetric image pixel S(.delta..sub.T, .theta..sub.T) from which the effects of the anterior segment retardance are removed…. FIG. 3 is an enlarged view of the macula centered on fovea 32, showing in greater detail the paths of the nerve fibers leaving fovea 32. The cell bodies 52 of the photoreceptor elements are in the very center of fovea 32. Cell bodies 52 send the axons called Henle fibers 54 to communicate with a ring of ganglion cells 56 surrounding fovea 32. Ganglion cells 56 in turn give rise to long axons of their own, constituting the retinal nerve fibers 48 that travel to optic nerve 34. Henle fibers 54 radiating from photoreceptor cell bodies 52 arranged precisely radially about the center of fovea 32. This precise radial array of Henle fibers 54, ending at the ring of ganglion cells 56, has an overall diameter subtending about four degrees of visual angle. Except for Henle fibers 54 in fovea 32, the only other retinal region having a radial array of nerve fibers is the area around the optic nerve head 50. Optic nerve head 50 subtends a visual angle of about five degrees. Both the Henle fibers and the other retinal nerve fibers are birefringent, with the slow optic axis (.theta.+.pi./2) of the birefringence being parallel to the direction of the fiber. [0046] FIG. 4 is a functional block diagram illustrating an exemplary embodiment 58 of a polarimeter suitable for use in the ophthalmological system of this invention. Polarimeter 58 is adapted to produce a plurality of pixels {S(.delta..sub.T, .theta..sub.T)} representing a two-dimensional polarimetric image suitable for use in analyzing a structure in eye 20 to provide, for example, an image map of the thickness of RNFL 40 or a polarimetric image of the retardance magnitude .delta..sub.H and orientation .theta..sub.H the Henle fiber layer 54. In FIG. 4, a laser diode 60 produces a linearly-polarized diagnostic optical signal 62, which is redirected by the polarizing beam splitter 64 to a non-polarizing beam splitter 66 and therefrom though the collimating lens 68 and the focusing lens 70 along an optical beam axis to the polygon scanner 72 and the galvo-mirror scanner 74. Scanners 72 and 74 provide a two-dimensional beam scan 76, each individual pixel of which has a linear polarization that is rotated by the half-wave plate 78 and the retarder 80, which may be embodied as a VCC or liquid crystal variable retarder (LCVR) or a fixed retarder or any useful combination of one or more thereof An output lens 82 steers the elements of two-dimensional beam scan 76 to the fundus 28 of eye 20. A moveable calibration test target 84 is used in cooperation with a CCD camera 86 and a fixation laser diode 88 (providing an optical fixation signal 90 that is transmitted along the optical beam axis) to automatically calibrate and orient the various elements of polarimeter 58 to eye 20. A reflected optical diagnostic signal 92 is returned from fundus 28 along the same optical path, to non-polarizing beam splitter 66, from whence it is transmitted through the pinhole 94 and the focusing lens 96 to the polarizing beam splitter 98. Polarizing beam splitter 98 separates the orthogonal polarization components 100 and 102, directing them respectively to the optical detectors 104 and 106. Operation of polarimeter 58 may be readily appreciated with reference to the above discussion the above-cited SLP patents included herein by reference. Not shown is the motor means required for independently rotating half-wave plate 78 about optical beam axis 90 to obtain the second polarimetric images required in accordance with this invention.).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra to utilize the cited limitations as suggested by Zhou. The suggestion/motivation for doing so would have been to enhance the speed and efficiency with accuracy for automated classification (see 0023). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra, while the teaching of Zhou continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 2, Kalra with Zhou teaches all elements as mentioned above in claim 1. Kalra further teaches a transfuse network configured to simultaneously process the raw RGB data (see 0019, 0125-0126, 0114, 0153). Kalra does not teach expressly birefringence output to provide nerve detection information.
Zhou, in the same field of endeavor, teaches birefringence output to provide nerve detection information (see 0027; 0045-0046; In one aspect, the invention is a method for analyzing the birefringence of a retinal structure of an eye including the steps of (a) producing a first polarimetric image of the retinal structure having a plurality of pixels {S(.delta..sub.T, .theta..sub.T)} each representing a measured retardance magnitude .delta..sub.T and orientation angle .theta..sub.T, (b) determining the fast axes of the pixel retardance angles {.theta..sub.T} corresponding to a biological feature of the retinal structure in the first polarimetric image, (c) determining anterior segment retardance magnitude .delta..sub.C and orientation .theta..sub.C values corresponding to the variation of the retardance magnitude .delta..sub.T over an annular region of the first polarimetric image, and (d) computing a second polarimetric image of the retinal structure having a plurality of pixels {S(.delta..sub.N, .theta..sub.N)} each corresponding to a first polarimetric image pixel S(.delta..sub.T, .theta..sub.T) from which the effects of the anterior segment retardance are removed…. FIG. 3 is an enlarged view of the macula centered on fovea 32, showing in greater detail the paths of the nerve fibers leaving fovea 32. The cell bodies 52 of the photoreceptor elements are in the very center of fovea 32. Cell bodies 52 send the axons called Henle fibers 54 to communicate with a ring of ganglion cells 56 surrounding fovea 32. Ganglion cells 56 in turn give rise to long axons of their own, constituting the retinal nerve fibers 48 that travel to optic nerve 34. Henle fibers 54 radiating from photoreceptor cell bodies 52 arranged precisely radially about the center of fovea 32. This precise radial array of Henle fibers 54, ending at the ring of ganglion cells 56, has an overall diameter subtending about four degrees of visual angle. Except for Henle fibers 54 in fovea 32, the only other retinal region having a radial array of nerve fibers is the area around the optic nerve head 50. Optic nerve head 50 subtends a visual angle of about five degrees. Both the Henle fibers and the other retinal nerve fibers are birefringent, with the slow optic axis (.theta.+.pi./2) of the birefringence being parallel to the direction of the fiber. [0046] FIG. 4 is a functional block diagram illustrating an exemplary embodiment 58 of a polarimeter suitable for use in the ophthalmological system of this invention. Polarimeter 58 is adapted to produce a plurality of pixels {S(.delta..sub.T, .theta..sub.T)} representing a two-dimensional polarimetric image suitable for use in analyzing a structure in eye 20 to provide, for example, an image map of the thickness of RNFL 40 or a polarimetric image of the retardance magnitude .delta..sub.H and orientation .theta..sub.H the Henle fiber layer 54. In FIG. 4, a laser diode 60 produces a linearly-polarized diagnostic optical signal 62, which is redirected by the polarizing beam splitter 64 to a non-polarizing beam splitter 66 and therefrom though the collimating lens 68 and the focusing lens 70 along an optical beam axis to the polygon scanner 72 and the galvo-mirror scanner 74. Scanners 72 and 74 provide a two-dimensional beam scan 76, each individual pixel of which has a linear polarization that is rotated by the half-wave plate 78 and the retarder 80, which may be embodied as a VCC or liquid crystal variable retarder (LCVR) or a fixed retarder or any useful combination of one or more thereof An output lens 82 steers the elements of two-dimensional beam scan 76 to the fundus 28 of eye 20. A moveable calibration test target 84 is used in cooperation with a CCD camera 86 and a fixation laser diode 88 (providing an optical fixation signal 90 that is transmitted along the optical beam axis) to automatically calibrate and orient the various elements of polarimeter 58 to eye 20. A reflected optical diagnostic signal 92 is returned from fundus 28 along the same optical path, to non-polarizing beam splitter 66, from whence it is transmitted through the pinhole 94 and the focusing lens 96 to the polarizing beam splitter 98. Polarizing beam splitter 98 separates the orthogonal polarization components 100 and 102, directing them respectively to the optical detectors 104 and 106. Operation of polarimeter 58 may be readily appreciated with reference to the above discussion the above-cited SLP patents included herein by reference. Not shown is the motor means required for independently rotating half-wave plate 78 about optical beam axis 90 to obtain the second polarimetric images required in accordance with this invention.).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra to utilize the cited limitations as suggested by Zhou. The suggestion/motivation for doing so would have been to enhance the speed and efficiency with accuracy for automated classification (see 0023). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra, while the teaching of Zhou continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 4, Kalra with Zhou teaches all elements as mentioned above in claim 1. Kalra further teaches raw polarized data (see 0063-0066, 0107-0109). Kalra does not teach expressly a birefringence map.
Zhou, in the same field of endeavor, teaches a birefringence map (see 0027-0029).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra with Zhou to utilize the cited limitations as suggested by Zhou. The suggestion/motivation for doing so would have been to enhance the speed and efficiency with accuracy for automated classification (see 0023). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra with Zhou, while the teaching of Zhou continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 12, Kalra teaches A method for nerve detection comprising: receiving at a general-purpose graphics processing unit (GPGPU), raw polarized data and raw RGB image data (see 0063, 0065, 0066, 0109, 0114, 0153; As a result, the polarization camera captures multiple input images 18 (or polarization raw frames) of the scene 1, where each of the polarization raw frames 18 corresponds to an image taken behind a polarization filter or polarizer at a different angle of polarization ϕ.sub.pol (e.g., 0 degrees, 45 degrees, 90 degrees, or 135 degrees). Each of the polarization raw frames is captured from substantially the same pose with respect to the scene 1 (e.g., the images captured with the polarization filter at 0 degrees, 45 degrees, 90 degrees, or 135 degrees are all captured by a same polarization camera located at a same location and orientation), as opposed to capturing the polarization raw frames from disparate locations and orientations with respect to the scene. The polarization camera 10 may be configured to detect light in a variety of different portions of the electromagnetic spectrum, such as the human-visible portion of the electromagnetic spectrum, red, green, and blue portions of the human-visible spectrum, as well as invisible portions of the electromagnetic spectrum such as infrared and ultraviolet… The polarization raw frames 18 are supplied to a processing circuit 100, described in more detail below, computes a segmentation map 20 based of the polarization raw frames 18. As shown in FIG. 1, in the segmentation map 20, the transparent objects 2 and the opaque objects 3 of the scene are all individually labeled, where the labels are depicted in FIG. 1 using different colors or patterns (e.g., vertical lines, horizontal lines, checker patterns, etc.), but where, in practice, each label may be represented by a different value (e.g., an integer value, where the different patterns shown in the figures correspond to different values) in the segmentation map….. Accordingly, extracting features such as polarization feature maps or polarization images from polarization raw frames 18 produces first tensors 50 from which transparent objects or other optically challenging objects such as translucent objects, multipath inducing objects, non-Lambertian objects, and non-reflective objects are more easily detected or separated from other objects in a scene. In some embodiments, the first tensors extracted by the feature extractor 800 may be explicitly derived features (e.g., hand crafted by a human designer) that relate to underlying physical phenomena that may be exhibited in the polarization raw frames (e.g., the calculation of AOLP and DOLP images, as discussed above). In some additional embodiments of the present disclosure, the feature extractor 800 extracts other non-polarization feature maps or non-polarization images, such as intensity maps for different colors of light (e.g., red, green, and blue light) and transformations of the intensity maps (e.g., applying image processing filters to the intensity maps). In some embodiments of the present disclosure the feature extractor 800 may be configured to extract one or more features that are automatically learned (e.g., features that are not manually specified by a human) through an end-to-end supervised training process based on labeled training data…. In some embodiments of the present disclosure, a same predictor or statistical model 900 is trained to detect both transparent objects and opaque objects (or to generate second tensors C in second representation space) based on training data containing labeled examples of both transparent objects and opaque objects. For example, in some such embodiments, a Polarized CNN architecture is used, such as the Polarized Mask R-CNN architecture shown in FIG. 9. In some embodiments, the Polarized Mask R-CNN architecture shown in FIG. 9 is further modified by adding one or more additional CNN backbones that compute one or more additional mode tensors. The additional CNN backbones may be trained based on additional first tensors. In some embodiments these additional first tensors include image maps computed based on color intensity images (e.g., intensity of light in different wavelengths, such as a red intensity image or color channel, a green intensity image or color channel, and a blue intensity image or color channel). In some embodiments, these additional first tensors include image maps computed based on combinations of color intensity images. In some embodiments, the fusion modules 920 fuse all of the mode tensors at each scale from each of the CNN backbones (e.g., including the additional CNN backbones); generating at the GPGPU, output based on the raw polarized data (see 0109; Accordingly, extracting features such as polarization feature maps or polarization images from polarization raw frames 18 produces first tensors 50 from which transparent objects or other optically challenging objects such as translucent objects, multipath inducing objects, non-Lambertian objects, and non-reflective objects are more easily detected or separated from other objects in a scene. In some embodiments, the first tensors extracted by the feature extractor 800 may be explicitly derived features (e.g., hand crafted by a human designer) that relate to underlying physical phenomena that may be exhibited in the polarization raw frames (e.g., the calculation of AOLP and DOLP images, as discussed above). In some additional embodiments of the present disclosure, the feature extractor 800 extracts other non-polarization feature maps or non-polarization images, such as intensity maps for different colors of light (e.g., red, green, and blue light) and transformations of the intensity maps (e.g., applying image processing filters to the intensity maps). In some embodiments of the present disclosure the feature extractor 800 may be configured to extract one or more features that are automatically learned (e.g., features that are not manually specified by a human) through an end-to-end supervised training process based on labeled training data.); and simultaneously processing the raw RGB data (see 0125-0128, 0153; In the embodiment shown in FIG. 9, derived feature maps 50 (e.g., including input polarization images such as AOLP ϕ and DOLP ρ images) are supplied as inputs to a Polarized CNN backbone 910. In the embodiment shown in FIG. 9, the input feature maps 50 include three input images: the intensity image (I) 52, the AOLP (P) 56, the DOLP (p) 54 from equation (1) as the input for detecting a transparent object and/or other optically challenging object. These images are computed from polarization raw frames 18 (e.g., images I.sub.0, I.sub.45, I.sub.90, and I.sub.135 as described above), normalized to be in a range (e.g., 8-bit values in the range [0-255]) and transformed into three-channel gray scale images to allow for easy transfer learning based on networks pre-trained on the MSCoCo dataset (see, e.g., Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.). [0126] In the embodiment shown in FIG. 9, each of the input derived feature maps 50 is supplied to a separate backbone: intensity B.sub.I(I) 912, AOLP backbone B.sub.ϕ(ϕ) 914, and DOLP backbone B.sub.ρ(φ 916. The CNN backbones 912, 914, and 916 compute tensors for each mode, or “mode tensors” (e.g., feature maps computed based on parameters learned during training or transfer learning of the CNN backbone, discussed in more detail below) C.sub.i,I, C.sub.i,ρ, C.sub.i,ϕ at different scales or resolutions i. While FIG. 9 illustrates an embodiment with five different scales i, embodiments of the present disclosure are not limited thereto and may also be applied to CNN backbones with different numbers of scales. [0127] Some aspects of embodiments of the present disclosure relate to a spatially-aware attention-fusion mechanism to perform multi-modal fusion (e.g., fusion of the feature maps computed from each of the different modes or different types of input feature maps, such as the intensity feature map I, the AOLP feature map ϕ, and the DOLP feature map ρ). [0128] For example, in the embodiment shown in FIG. 9, the mode tensors C.sub.i,I, C.sub.i,ρ, C.sub.i,ϕ (tensors for each mode) computed from corresponding backbones B.sub.I, B.sub.ρ, B.sub.ϕ at each scale i are fused using fusion layers 922, 923, 924, 925 (collectively, fusion layers 920) for corresponding scales. For example, fusion layer 922 is configured to fuse mode tensors C.sub.2,I, C.sub.2,ρ, C.sub.2,ϕ computed at scale i=2 to compute a fused tensor C.sub.2. Likewise, fusion layer 923 is configured to fuse mode tensors C.sub.3,I, C.sub.3,ρ, C.sub.3,ϕ computed at scale i=3 to compute a fused tensor C.sub.3, and similar computations may be performed by fusion layers 924 and 925 to compute fused feature maps C.sub.4 and C.sub.5, respectively, based on respective mode tensors for their scales. The fused tensors C.sub.i (e.g., C.sub.2, C.sub.3, C.sub.4, C.sub.5), or second tensors, such as fused feature maps, computed by the fusion layers 920 are then supplied as input to a prediction module 950, which is configured to compute a prediction from the fused tensors, where the prediction may be an output such as a segmentation map 20, a classification, a textual description, or the like…. In some embodiments of the present disclosure, a same predictor or statistical model 900 is trained to detect both transparent objects and opaque objects (or to generate second tensors C in second representation space) based on training data containing labeled examples of both transparent objects and opaque objects. For example, in some such embodiments, a Polarized CNN architecture is used, such as the Polarized Mask R-CNN architecture shown in FIG. 9. In some embodiments, the Polarized Mask R-CNN architecture shown in FIG. 9 is further modified by adding one or more additional CNN backbones that compute one or more additional mode tensors. The additional CNN backbones may be trained based on additional first tensors. In some embodiments these additional first tensors include image maps computed based on color intensity images (e.g., intensity of light in different wavelengths, such as a red intensity image or color channel, a green intensity image or color channel, and a blue intensity image or color channel). In some embodiments, these additional first tensors include image maps computed based on combinations of color intensity images. In some embodiments, the fusion modules 920 fuse all of the mode tensors at each scale from each of the CNN backbones (e.g., including the additional CNN backbones).). Kalra does not teach expressly a birefringence and the birefringence output to provide nerve detection information.
Zhou, in the same field of endeavor, teaches a birefringence and the birefringence output to provide nerve detection information (see 0027; 0045-0046; In one aspect, the invention is a method for analyzing the birefringence of a retinal structure of an eye including the steps of (a) producing a first polarimetric image of the retinal structure having a plurality of pixels {S(.delta..sub.T, .theta..sub.T)} each representing a measured retardance magnitude .delta..sub.T and orientation angle .theta..sub.T, (b) determining the fast axes of the pixel retardance angles {.theta..sub.T} corresponding to a biological feature of the retinal structure in the first polarimetric image, (c) determining anterior segment retardance magnitude .delta..sub.C and orientation .theta..sub.C values corresponding to the variation of the retardance magnitude .delta..sub.T over an annular region of the first polarimetric image, and (d) computing a second polarimetric image of the retinal structure having a plurality of pixels {S(.delta..sub.N, .theta..sub.N)} each corresponding to a first polarimetric image pixel S(.delta..sub.T, .theta..sub.T) from which the effects of the anterior segment retardance are removed…. FIG. 3 is an enlarged view of the macula centered on fovea 32, showing in greater detail the paths of the nerve fibers leaving fovea 32. The cell bodies 52 of the photoreceptor elements are in the very center of fovea 32. Cell bodies 52 send the axons called Henle fibers 54 to communicate with a ring of ganglion cells 56 surrounding fovea 32. Ganglion cells 56 in turn give rise to long axons of their own, constituting the retinal nerve fibers 48 that travel to optic nerve 34. Henle fibers 54 radiating from photoreceptor cell bodies 52 arranged precisely radially about the center of fovea 32. This precise radial array of Henle fibers 54, ending at the ring of ganglion cells 56, has an overall diameter subtending about four degrees of visual angle. Except for Henle fibers 54 in fovea 32, the only other retinal region having a radial array of nerve fibers is the area around the optic nerve head 50. Optic nerve head 50 subtends a visual angle of about five degrees. Both the Henle fibers and the other retinal nerve fibers are birefringent, with the slow optic axis (.theta.+.pi./2) of the birefringence being parallel to the direction of the fiber. [0046] FIG. 4 is a functional block diagram illustrating an exemplary embodiment 58 of a polarimeter suitable for use in the ophthalmological system of this invention. Polarimeter 58 is adapted to produce a plurality of pixels {S(.delta..sub.T, .theta..sub.T)} representing a two-dimensional polarimetric image suitable for use in analyzing a structure in eye 20 to provide, for example, an image map of the thickness of RNFL 40 or a polarimetric image of the retardance magnitude .delta..sub.H and orientation .theta..sub.H the Henle fiber layer 54. In FIG. 4, a laser diode 60 produces a linearly-polarized diagnostic optical signal 62, which is redirected by the polarizing beam splitter 64 to a non-polarizing beam splitter 66 and therefrom though the collimating lens 68 and the focusing lens 70 along an optical beam axis to the polygon scanner 72 and the galvo-mirror scanner 74. Scanners 72 and 74 provide a two-dimensional beam scan 76, each individual pixel of which has a linear polarization that is rotated by the half-wave plate 78 and the retarder 80, which may be embodied as a VCC or liquid crystal variable retarder (LCVR) or a fixed retarder or any useful combination of one or more thereof An output lens 82 steers the elements of two-dimensional beam scan 76 to the fundus 28 of eye 20. A moveable calibration test target 84 is used in cooperation with a CCD camera 86 and a fixation laser diode 88 (providing an optical fixation signal 90 that is transmitted along the optical beam axis) to automatically calibrate and orient the various elements of polarimeter 58 to eye 20. A reflected optical diagnostic signal 92 is returned from fundus 28 along the same optical path, to non-polarizing beam splitter 66, from whence it is transmitted through the pinhole 94 and the focusing lens 96 to the polarizing beam splitter 98. Polarizing beam splitter 98 separates the orthogonal polarization components 100 and 102, directing them respectively to the optical detectors 104 and 106. Operation of polarimeter 58 may be readily appreciated with reference to the above discussion the above-cited SLP patents included herein by reference. Not shown is the motor means required for independently rotating half-wave plate 78 about optical beam axis 90 to obtain the second polarimetric images required in accordance with this invention.).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra to utilize the cited limitations as suggested by Zhou. The suggestion/motivation for doing so would have been to enhance the speed and efficiency with accuracy for automated classification (see 0023). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra, while the teaching of Zhou continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 13, the claim is analyzed as a method that implements the limitations of claim 2 (see rejection of claim 2).
Regarding claim 15, the claim is analyzed as a method that implements the limitations of claim 4 (see rejection of claim 4).
Claims 10-11, 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Kalra et al (US 2021/0264607 A1) with Zhou et al (US 2004/0061834 A1), and further in view of Feng et al (IEEE: “Multi-Modal Transformer for Accelerated MR Imaging”).
Regarding claims 10-11, Kalra with Zhou teaches all elements as mentioned above in claim 1. Kalra further teaches an RGB modality of the RGB image (see 0114, 0153). Kalra with Zhou does not teach a transformer block that fuses a birefringence modality of the birefringence output; a birefringence map based on the fused birefringence modality and RGB modality.
Zhou, in the same field of endeavor, teaches a birefringence modality of the birefringence output (see 0018, 0027-0029, 0045-0046) a birefringence map (see 0027-0029, 0046).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra with Zhou to utilize the cited limitations as suggested by Zhou. The suggestion/motivation for doing so would have been to enhance the speed and efficiency with accuracy for automated classification (see 0023). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra with Zhou, while the teaching of Zhou continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Feng, in the same field of endeavor, teaches a transformer block that fuses (see section IV).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra with Zhou to utilize the cited limitations as suggested by Feng. The suggestion/motivation for doing so would have been to reduce artifacts (see conclusion). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra with Zhou, while the teaching of Feng continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claims 21-22, the claim is analyzed as a method that implements the limitations of claims 10-11 (see rejection of claims 10-11).
Claims 3, 5, 6, 14, 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Kalra et al (US 2021/0264607 A1) with Zhou et al (US 2004/0061834 A1), and further in view of Mordechai et al (US 2021/0174528 A1).
Regarding claims 3, 5, Kalra with Zhou teaches all elements as mentioned above in claim 1. Kalra with Zhou does not teach a dual RGB and polarimetric imaging device, said GPGPU receiving the raw polarized data and the raw RGB data from said image acquisition device; a deep learning network trained on domain-specific data to optimize the deep learning network to produce a fast inference network output.
Mordechai, in the same field of endeavor, teaches a dual RGB and polarimetric imaging device, said GPGPU receiving the raw polarized data and the raw RGB data from said image acquisition device (see 0038, 0039, 0046); a deep learning network trained on domain-specific data to optimize the deep learning network to produce a fast inference network output (see 0047, 0046-0047, 0035-0036).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra with Zhou to utilize the cited limitations as suggested by Mordechai. The suggestion/motivation for doing so would have been to further enhance depth estimation (see 0035). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra with Zhou, while the teaching of Zhou continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claims 14, 16, the claims are analyzed as a method that implements the limitations of claims 3 and 5. (see rejection of claims 3 and 5).
Regarding claim 6, Kalra, Zhou with Mordechai teaches all elements as mentioned above in claim 3. Karla teaches the segmentation mask (see 0128, 0133, 0134). Kalra, Zhou with Mordechai does not teach nerve.
Zhou, in the same field of endeavor, teaches nerve (see 0018, 0044-0046).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra, Zhou with Mordechai to utilize the cited limitations as suggested by Zhou. The suggestion/motivation for doing so would have been to enhance the speed and efficiency with accuracy for automated classification (see 0023). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra, Zhou with Mordechai, while the teaching of Zhou continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Regarding claim 17, the claims are analyzed as a method that implements the limitations of claim 6. (see rejection of claim 6).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Kalra et al (US 2021/0264607 A1) with Zhou et al (US 2004/0061834 A1), and further in view of Cha et al (BOE: “Real-time, label-free, intraoperative visualization of peripheral nerves and micro vasculatures using multimodal optical imaging techniques”).
Regarding claim 19, Kalra with Zhou teaches all elements as mentioned above in claim 12. Kalra with Zhou does not teach uses a BRF representation of the birefringence output and the RGB image data to identify nerve structure.
Cha, in the same field of endeavor, teaches uses a BRF representation of the birefringence output and the RGB image data to identify nerve structure (see section 2.2, 3.1).
It would have been obvious (before the effective filing date of the claimed invention) or (at the time the invention was made) to one of ordinary skill in the art to modify Kalra with Zhou to utilize the cited limitations as suggested by Cha. The suggestion/motivation for doing so would have been to enables noninvasive visualization of critical anatomic structures during surgical dissection (see abstract). Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kalra with Zhou, while the teaching of Cha continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Allowable Subject Matter
Claims 7-9, 18, 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claims 7-9, none of the references of record alone or in combination suggest or fairly teach wherein the raw polarized data has a plurality of output pixels, and said GPGPU performs birefringence calculations in parallel for each output pixel of the plurality of output pixels, to obtain a birefringence output.
Regarding claim 18, none of the references of record alone or in combination suggest or fairly teach wherein the raw polarized data has a plurality of output pixels, and the GPGPU performs birefringence calculations in parallel for each output pixel of the plurality of output pixels, to obtain a birefringence output.
Regarding claim 20, none of the references of record alone or in combination suggest or fairly teach wherein the GPGPU performs birefringence mapping on the raw polarized data, normalizes the birefringence mapping and applies a BRF colormap to obtain the BRF representation.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EDWARD PARK. The examiner’s contact information is as follows:
Telephone: (571)270-1576 | Fax: 571.270.2576 | Edward.Park@uspto.gov
For email communications, please notate MPEP 502.03, which outlines procedures pertaining to communications via the internet and authorization. A sample authorization form is cited within MPEP 502.03, section II.
The examiner can normally be reached on M-F 9-6 CST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M. Villecco, can be reached on (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EDWARD PARK/Primary Examiner, Art Unit 2661
Prosecution Insights