DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
The amendments to claims 1-5, 8-12, 15-19 are accepted and entered.
Claims 1-20 are pending regarding this application.
Response to Arguments
Applicant’s arguments, see Remarks, filed 08/18/2026, with respect to the Claim Objections applied to claims 1, 3, 4, 8, 10, 11, 15, 17, and 18 have been fully considered and are persuasive. Applicant has sufficiently amended the above claims, as shown in the Claims filed on 08/18/2026, in accordance with the suggestions outline in the Non-Final Rejection mailed on 05/21/2026. The Claim Objections of claims 1, 3, 4, 8, 10, 11, 15, 17, and 18 has been withdrawn.
Applicant’s arguments, see Remarks, filed 08/18/2026, with respect to the 112(b) Rejections applied to claims 1-20 have been fully considered and are persuasive. Applicant has sufficiently amended claims 1, 5, 8, 12, and 15, and 19, as shown in the Claims filed on 08/18/2026, to overcome the issues as outlined in the Non-Final Rejection mailed on 05/21/2026. The 112(b) Rejections of claims 1-20 has been withdrawn.
Applicant’s arguments with respect to the 103 Rejection(s) applied to independent claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. It should be noted that the examiner disagrees that Brewington et al. fails to teach the first and second modalities as taught in amended claim 1. Examiner would like to point out para. [0071] as specifically teaching multiple (different) sensors capable of acquiring low resolution sensor images and high resolution sensor images, respectively. However, additional prior art has been added that similarly teaches the first and second imaging modalities, along with additionally added subject matter within claim 1. See the inclusion of Eggers et al. (U.S. Publication No. 2009/0102858 A1) as shown in the 103 rejection below regarding this matter. Please additionally note that Phillips (U.S. Publication No. 2024/0328815 A1) was another reference considered to be of particular relevance, as it additionally teaches both the first and second modalities and merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 5, 8, 9, 11, 12, 15, 16, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (U.S. Publication No. 2021/0133956 A1) in view of Brewington et al. (U.S. Publication No. 2021/0027112 A1), hereinafter Brewington and Eggers et al. (U.S. Publication No. 2009/0102858 A1), hereinafter Eggers.
Regarding claim 1, Chen teaches a computer-implemented method for efficient analysis of image data (Chen teaches that “the present disclosure may be more computationally efficient than existing methods and can classify, detect, and segment lesions on microscopy images in real time” in para. [0057]), the method comprising:
retrieving an image of an area of interest with a first resolution, wherein said image comprises image data corresponding to a first imaging modality (Chen teaches an input tissue image at a first resolution in para. [0045]. This input tissue image is interpreted as equivalent to the received image of an area of interest with a first resolution. Additionally, the input image is captured with a camera as shown in para. [0019]);
performing a first image analysis on said image with said first resolution to generate a first image analysis result and a first uncertainty map for said image with said first resolution (Chen teaches “performing a first image analysis at a first resolution of an input tissue image” and “generating an uncertainty map based on performing the first image analysis” as shown in para. [0045]-[0046] and FIG. 4 #410 and #420);
identifying an area in said first uncertainty map with an uncertainty (Chen teaches “identifying a set of uncertain regions of the input tissue image based on the uncertainty map” in para. [0047] and FIG. 4 $430);
retrieving a segment of said image with a second resolution corresponding to said area in said first uncertainty map (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” as shown in para. [0048] and FIG. 4 #440);
performing a second image analysis on said segment of said image with said second resolution to generate a second image analysis result (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” as shown in para. [0048] and FIG. 4 #440) and a second uncertainty map for said segment of said image with said second resolution (Chen teaches “If there are uncertain regions, the image patches of uncertain regions in higher resolution may be computed. The system may iterate this process until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” as shown in para. [0053] and FIG. 5. This iterative process inherently involves generating a second uncertainty map relating to the segment of the image with the second resolution); and
merging said second image analysis result into said first image analysis result (Chen teaches “generating an analysis result based on the first image analysis and the second image analysis” in para. [0049] and FIG. 4 #450. See also that “the results and the uncertainty map generated in each resolution may be fused to generate the final results and the final uncertainty map” as shown in para. [0053] and FIG. 5. Here, within the first uncertainty map with the first image analysis result, uncertain regions are identified and the image patches of uncertain regions in higher resolution may be computed. Here, these uncertain regions with higher resolutions are equivalent to the claimed second uncertainty map/second image analysis result, and are merged with the first analysis result as shown in para. [0053]).
While Chen teaches “the system may iterate this process until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” in para. [0053], Chen fails to teach identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold, wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality and merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (emphasis added).
However, Brewington teaches wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality (Brewington teaches para. [0033] wherein Brewington recites that the environment may include one or more sensors, wherein the sensors may be optical and/or thermal sensors, and, “with respect to the resolution of the one or more sensors, the semantic state system can determine one or more sensor resolutions of the one or more sensors (e.g., accessing sensor type data including the sensor resolution for a sensor)”. Here, the optical sensor(s) may be interpreted as the first imaging modality while the thermal sensor(s) may be interpreted as the second imaging modality) and identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold (Brewington teaches “selectively obtaining updated sensor data from the high resolution sensor when the uncertainty level [for sensor data captured with a low resolution sensor] exceeds an uncertainty level threshold” in para. [0021]. See also that “the uncertainty level determined for the most recent data provided by the target sensor may be dependent on the sensor resolutions of the one or more sensors not including the target sensor” from para. [0033]).
Chen and Brewington are both considered to be analogous to the claimed invention because they are in the same field of using uncertainty to efficiently collect and analyze image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen to incorporate the teachings of Brewington and include “identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold, wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality”. The motivation for doing so would have been that, “as obtaining and processing data from a higher resolution sensor can require a greater amount of resource utilization in comparison to a lower resolution sensor, the resource utilization required to obtain sensor data from the one or more sensors can be made more efficient by obtaining sensor data from higher resolution sensors less frequently than lower resolution sensors”, as suggested by Brewington in para. [0021]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen with Brewington to obtain the invention specified in the above claim limitations.
Chen and Brewington fail to teach merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (emphasis added).
However, similar to Brewington, Eggers additionally teaches wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality (Eggers teaches image data corresponding to a first imaging modality, wherein (first) image data may be captured using an infrared and/or visible light camera as shown in para. [0027]-[0028]. Eggers additionally teaches utilizing (second) image data from a second modality, wherein the second modality may be a high-resolution image-generating radar system as shown in para. [0027]). Eggers’s teaching of the first and second imaging modality is referenced here in order to provide a teaching of the first and second imaging modality which more closely aligns with the specific first and second imaging modality as described by the applicant in para. [0063] of the specification wherein the images output from the two or more imaging modalities are RGB images and radar images.
Eggers further teaches merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (Eggers teaches “replac[ing] an image region of the first type in an infrared image with a conditioned information item from a radar system, in particular a high-resolution (image-generating) radar system” in para. [0027]. Here, this teaching of Eggers can be combined with Chen’s teaching of merging the first image analysis result and the second image analysis result to teach the above limitation).
Chen, Brewington, and Eggers are all considered to be analogous to the claimed invention because they are in the same field of using uncertainty to efficiently collect and analyze image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington) to incorporate the teachings of Eggers and include “merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result”. The motivation for doing so would have been that “it therefore becomes possible to display clearly, in the image regions of the first type, not only the objects which are to be highlighted but also further objects, which may possibly be relevant, on the image display” and “to make use of the data from other sensors or sensor streams and to substitute them for the image data originally supplied by the camera of the image display system”, as suggested by Eggers in para. [0025] and [0027], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen and Brewington with Eggers to obtain the invention specified in claim 1.
Regarding claim 2, Chen, Brewington, and Eggers teach the method as recited in claim 1,
wherein the image data corresponding to said first imaging modality comprises red, green, and blue image data (Eggers teaches utilizing a color camera to capture image data as shown in para. [0027]), and wherein the image data corresponding to said second imaging modality comprises radar image data (Eggers teaches utilizing radar system capable of generating images at a high resolution which is interpreted as equivalent to the claimed second imaging modality as shown in para. [0027]. Here, the teachings of Eggers can be combined with Brewington’s teaching of the first and second imaging modality, and Chen’s teaching of the image data as shown in claim 1 to teach the above subject matter of claim 2).
Similar motivations as applied to claim 1 can be applied here to claim 2.
Regarding claim 4, Chen, Brewington, and Eggers teach the method as recited in claim 1,
wherein said first image analysis performed on said image with said first resolution and said second image analysis performed on said segment of said image with said second resolution generate respective pixel-wise classifications (Chen teaches that “the CNN may be configured to directly generate region proposal, segmentation, and uncertainty maps. Segmentation classifies an image on a pixel-by-pixel basis” as shown in para. [0054] and FIG. 6).
Regarding claim 5, Chen, Brewington, and Eggers teach the method as recited in claim 1 further comprising:
identifying a further area in said second uncertainty map for said segment of said image with said second resolution having an uncertainty beyond said defined uncertainty threshold; and retrieving a further segment of said image with a third resolution corresponding to said further area, wherein said third resolution has a higher resolution than said second resolution (Chen teaches that “the system may iterate [the] process [of identifying uncertain regions and computing patches of uncertain regions in higher resolution] until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” in para. [0053]. Here, since the process can be repeated multiple times, it is inherent that there exists at least a third resolution which has a higher resolution than the second resolution. Chen’s teaching of this iterative process can be combined with Brewington’s teaching of “selectively obtaining updated sensor data from the high resolution sensor when the uncertainty level [for sensor data captured with a low resolution sensor] exceeds an uncertainty level threshold” in para. [0021] to teach the above limitations).
Similar motivation as applied to claim 1 can be applied here to claim 5.
Regarding claim 8, Chen teaches a computer program product for efficient analysis of image data, the computer program product comprising one or more computer readable storage mediums having program code embodied therewith (Chen teaches that the method may be implemented on a “virtual machine 224-2 [that] includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine” in para. [0031]. Chen additionally teaches a computer readable storage medium storing instruction for use by a processor in para. [0037]. See also para. [0041]-[0042]), the program code comprising programming instructions for:
retrieving an image of an area of interest with a first resolution, wherein said image comprises image data corresponding to a first imaging modality (Chen teaches an input tissue image at a first resolution in para. [0045]. This input tissue image is interpreted as equivalent to the received image of an area of interest with a first resolution. Additionally, the input image is captured with a camera as shown in para. [0019]);
performing a first image analysis on said image with said first resolution to generate a first image analysis result and a first uncertainty map for said image with said first resolution (Chen teaches “performing a first image analysis at a first resolution of an input tissue image” and “generating an uncertainty map based on performing the first image analysis” as shown in para. [0045]-[0046] and FIG. 4 #410 and #420);
identifying an area in said first uncertainty map with an uncertainty (Chen teaches “identifying a set of uncertain regions of the input tissue image based on the uncertainty map” in para. [0047] and FIG. 4 $430);
retrieving a segment of said image with a second resolution corresponding to said area in said first uncertainty map (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” as shown in para. [0048] and FIG. 4 #440);
performing a second image analysis on said segment of said image with said second resolution to generate a second image analysis result (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” as shown in para. [0048] and FIG. 4 #440) and a second uncertainty map for said segment of said image with said second resolution (Chen teaches “If there are uncertain regions, the image patches of uncertain regions in higher resolution may be computed. The system may iterate this process until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” as shown in para. [0053] and FIG. 5. This iterative process inherently involves generating a second uncertainty map relating to the segment of the image with the second resolution); and
merging said second image analysis result into said first image analysis result (Chen teaches “generating an analysis result based on the first image analysis and the second image analysis” in para. [0049] and FIG. 4 #450. See also that “the results and the uncertainty map generated in each resolution may be fused to generate the final results and the final uncertainty map” as shown in para. [0053] and FIG. 5. Here, within the first uncertainty map with the first image analysis result, uncertain regions are identified and the image patches of uncertain regions in higher resolution may be computed. Here, these uncertain regions with higher resolutions are equivalent to the claimed second uncertainty map/second image analysis result, and are merged with the first analysis result as shown in para. [0053]).
While Chen teaches “the system may iterate this process until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” in para. [0053], Chen fails to teach identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold, wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality and merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (emphasis added).
However, Brewington teaches wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality (Brewington teaches collecting lower resolution sensor data from a first sensor 402, and higher resolution sensor data from a second sensor 404 in para. [0071]) and identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold (Brewington teaches “selectively obtaining updated sensor data from the high resolution sensor when the uncertainty level [for sensor data captured with a low resolution sensor] exceeds an uncertainty level threshold” in para. [0021]).
Chen and Brewington are both considered to be analogous to the claimed invention because they are in the same field of using uncertainty to efficiently collect and analyze image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen to incorporate the teachings of Brewington and include “identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold, wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality”. The motivation for doing so would have been that, “as obtaining and processing data from a higher resolution sensor can require a greater amount of resource utilization in comparison to a lower resolution sensor, the resource utilization required to obtain sensor data from the one or more sensors can be made more efficient by obtaining sensor data from higher resolution sensors less frequently than lower resolution sensors”, as suggested by Brewington in para. [0021]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen with Brewington to obtain the invention specified in the above claim limitations.
Chen and Brewington fail to teach merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (emphasis added).
However, similar to Brewington, Eggers additionally teaches wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality (Eggers teaches image data corresponding to a first imaging modality, wherein (first) image data may be captured using a infrared and/or visible light camera as shown in para. [0027]-[0028]. Eggers additionally teaches utilizing (second) image data from a second modality, wherein the second modality may be a high-resolution image-generating radar system as shown in para. [0027]). Eggers further teaches merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (Eggers teaches “replac[ing] an image region of the first type in an infrared image with a conditioned information item from a radar system, in particular a high-resolution (image-generating) radar system” in para. [0027]. Here, this teaching of Eggers can be combined with Chen’s teaching of merging the first image analysis result and the second image analysis result to teach the above limitation).
Chen, Brewington, and Eggers are all considered to be analogous to the claimed invention because they are in the same field of using uncertainty to efficiently collect and analyze image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington) to incorporate the teachings of Eggers and include “merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result”. The motivation for doing so would have been that “it therefore becomes possible to display clearly, in the image regions of the first type, not only the objects which are to be highlighted but also further objects, which may possibly be relevant, on the image display” and “to make use of the data from other sensors or sensor streams and to substitute them for the image data originally supplied by the camera of the image display system”, as suggested by Eggers in para. [0025] and [0027], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen and Brewington with Eggers to obtain the invention specified in claim 8.
Regarding claim 9, Chen, Brewington, and Eggers teach the computer program product as recited in claim 8,
wherein the image data corresponding to said first imaging modality comprises red, green, and blue image data (Eggers teaches utilizing a color camera to capture image data as shown in para. [0027]), and wherein the image data corresponding to said second imaging modality comprises radar image data (Eggers teaches utilizing radar system capable of generating images at a high resolution which is interpreted as equivalent to the claimed second imaging modality as shown in para. [0027]. Here, the teachings of Eggers can be combined with Brewington’s teaching of the first and second imaging modality, and Chen’s teaching of the image data as shown in claim 8 to teach the above subject matter of claim 9).
Similar motivations as applied to claim 8 can be applied here to claim 9.
Regarding claim 11, Chen, Brewington, and Eggers teach the computer program product as recited in claim 8,
wherein said first image analysis performed on said image with said first resolution and said second image analysis performed on said segment of said image with said second resolution generate respective pixel-wise classifications (Chen teaches that “the CNN may be configured to directly generate region proposal, segmentation, and uncertainty maps. Segmentation classifies an image on a pixel-by-pixel basis” as shown in para. [0054] and FIG. 6).
Regarding claim 12, Chen, Brewington, and Eggers teach the computer program product as recited in claim 8, wherein the program code further comprises the programming instructions for:
identifying a further area in said second uncertainty map for said segment of said image with said second resolution having an uncertainty beyond said defined uncertainty threshold; and retrieving a further segment of said image with a third resolution corresponding to said further area, wherein said third resolution has a higher resolution than said second resolution (Chen teaches that “the system may iterate [the] process [of identifying uncertain regions and computing patches of uncertain regions in higher resolution] until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” in para. [0053]. Here, since the process can be repeated multiple times, it is inherent that there exists at least a third resolution which has a higher resolution than the second resolution. Chen’s teaching of this iterative process can be combined with Brewington’s teaching of “selectively obtaining updated sensor data from the high resolution sensor when the uncertainty level [for sensor data captured with a low resolution sensor] exceeds an uncertainty level threshold” in para. [0021] to teach the above limitations).
Similar motivation as applied to claim 8 can be applied here to claim 12.
Regarding claim 15, Chen teaches a system (Chen, FIG. 5), comprising:
a memory for storing a computer program for efficient analysis of image data (Chen teaches “device 300 may perform these processes in response to processor 320 executing software instructions stored by a non-transitory computer-readable medium, such as memory 330 and/or storage component 340” in para. [0041]. See also the system as taught in para. [0031]); and
a processor connected to said memory, wherein said processor is configured to execute program instructions of the computer program (Chen teaches “when executed, software instructions stored in memory 330 and/or storage component 340 may cause processor 320 to perform one or more processes described herein” in para. [0042]) comprising:
retrieving an image of an area of interest with a first resolution, wherein said image comprises image data corresponding to a first imaging modality (Chen teaches an input tissue image at a first resolution in para. [0045]. This input tissue image is interpreted as equivalent to the received image of an area of interest with a first resolution. Additionally, the input image is captured with a camera as shown in para. [0019]);
performing a first image analysis on said image with said first resolution to generate a first image analysis result and a first uncertainty map for said image with said first resolution (Chen teaches “performing a first image analysis at a first resolution of an input tissue image” and “generating an uncertainty map based on performing the first image analysis” as shown in para. [0045]-[0046] and FIG. 4 #410 and #420);
identifying an area in said first uncertainty map with an uncertainty (Chen teaches “identifying a set of uncertain regions of the input tissue image based on the uncertainty map” in para. [0047] and FIG. 4 $430);
retrieving a segment of said image with a second resolution corresponding to said area in said first uncertainty map (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” as shown in para. [0048] and FIG. 4 #440);
performing a second image analysis on said segment of said image with said second resolution to generate a second image analysis result (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” as shown in para. [0048] and FIG. 4 #440) and a second uncertainty map for said segment of said image with said second resolution (Chen teaches “If there are uncertain regions, the image patches of uncertain regions in higher resolution may be computed. The system may iterate this process until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” as shown in para. [0053] and FIG. 5. This iterative process inherently involves generating a second uncertainty map relating to the segment of the image with the second resolution); and
merging said second image analysis result into said first image analysis result (Chen teaches “generating an analysis result based on the first image analysis and the second image analysis” in para. [0049] and FIG. 4 #450. See also that “the results and the uncertainty map generated in each resolution may be fused to generate the final results and the final uncertainty map” as shown in para. [0053] and FIG. 5. Here, within the first uncertainty map with the first image analysis result, uncertain regions are identified and the image patches of uncertain regions in higher resolution may be computed. Here, these uncertain regions with higher resolutions are equivalent to the claimed second uncertainty map/second image analysis result, and are merged with the first analysis result as shown in para. [0053]).
While Chen teaches “the system may iterate this process until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” in para. [0053], Chen fails to teach identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold, wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality and merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (emphasis added).
However, Brewington teaches wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality (Brewington teaches collecting lower resolution sensor data from a first sensor 402, and higher resolution sensor data from a second sensor 404 in para. [0071]) and identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold (Brewington teaches “selectively obtaining updated sensor data from the high resolution sensor when the uncertainty level [for sensor data captured with a low resolution sensor] exceeds an uncertainty level threshold” in para. [0021]).
Chen and Brewington are both considered to be analogous to the claimed invention because they are in the same field of using uncertainty to efficiently collect and analyze image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen to incorporate the teachings of Brewington and include “identifying an area in said first uncertainty map with an uncertainty beyond a defined uncertainty threshold, wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality”. The motivation for doing so would have been that, “as obtaining and processing data from a higher resolution sensor can require a greater amount of resource utilization in comparison to a lower resolution sensor, the resource utilization required to obtain sensor data from the one or more sensors can be made more efficient by obtaining sensor data from higher resolution sensors less frequently than lower resolution sensors”, as suggested by Brewington in para. [0021]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen with Brewington to obtain the invention specified in the above claim limitations.
Chen and Brewington fail to teach merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (emphasis added).
However, similar to Brewington, Eggers additionally teaches wherein said segment comprises image data corresponding to said first imaging modality and image data corresponding to a second imaging modality (Eggers teaches image data corresponding to a first imaging modality, wherein (first) image data may be captured using a infrared and/or visible light camera as shown in para. [0027]-[0028]. Eggers additionally teaches utilizing (second) image data from a second modality, wherein the second modality may be a high-resolution image-generating radar system as shown in para. [0027]). Eggers further teaches merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result (Eggers teaches “replac[ing] an image region of the first type in an infrared image with a conditioned information item from a radar system, in particular a high-resolution (image-generating) radar system” in para. [0027]. Here, this teaching of Eggers can be combined with Chen’s teaching of merging the first image analysis result and the second image analysis result to teach the above limitation).
Chen, Brewington, and Eggers are all considered to be analogous to the claimed invention because they are in the same field of using uncertainty to efficiently collect and analyze image data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington) to incorporate the teachings of Eggers and include “merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result”. The motivation for doing so would have been that “it therefore becomes possible to display clearly, in the image regions of the first type, not only the objects which are to be highlighted but also further objects, which may possibly be relevant, on the image display” and “to make use of the data from other sensors or sensor streams and to substitute them for the image data originally supplied by the camera of the image display system”, as suggested by Eggers in para. [0025] and [0027], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen and Brewington with Eggers to obtain the invention specified in claim 15.
Regarding claim 16, Chen, Brewington, and Eggers teach the system as recited in claim 15,
wherein the image data corresponding to said first imaging modality comprises red, green, and blue image data (Eggers teaches utilizing a color camera to capture image data as shown in para. [0027]), and wherein the image data corresponding to said second imaging modality comprises radar image data (Eggers teaches utilizing radar system capable of generating images at a high resolution which is interpreted as equivalent to the claimed second imaging modality as shown in para. [0027]. Here, the teachings of Eggers can be combined with Brewington’s teaching of the first and second imaging modality, and Chen’s teaching of the image data as shown in claim 15 to teach the above subject matter of claim 16).
Similar motivations as applied to claim 15 can be applied here to claim 16.
Regarding claim 18, Chen, Brewington, and Eggers teach the system as recited in claim 15,
wherein said first image analysis performed on said image with said first resolution and said second image analysis performed on said segment of said image with said second resolution generate respective pixel-wise classifications (Chen teaches that “the CNN may be configured to directly generate region proposal, segmentation, and uncertainty maps. Segmentation classifies an image on a pixel-by-pixel basis” as shown in para. [0054] and FIG. 6).
Regarding claim 19, Chen, Brewington, and Eggers teach the system as recited in claim 15, wherein the program instructions of the computer program further comprise:
identifying a further area in said second uncertainty map for said segment of said image with said second resolution having an uncertainty beyond said defined uncertainty threshold; and retrieving a further segment of said image with a third resolution corresponding to said further area, wherein said third resolution has a higher resolution than said second resolution (Chen teaches that “the system may iterate [the] process [of identifying uncertain regions and computing patches of uncertain regions in higher resolution] until a preset condition is satisfied (e.g., no further refinement is necessary, no uncertain regions are left, the highest resolution has been reached, etc.)” in para. [0053]. Here, since the process can be repeated multiple times, it is inherent that there exists at least a third resolution which has a higher resolution than the second resolution. Chen’s teaching of this iterative process can be combined with Brewington’s teaching of “selectively obtaining updated sensor data from the high resolution sensor when the uncertainty level [for sensor data captured with a low resolution sensor] exceeds an uncertainty level threshold” in para. [0021] to teach the above limitations).
Similar motivation as applied to claim 15 can be applied here to claim 19.
Claims 3, 10, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (U.S. Publication No. 2021/0133956 A1) in view of Brewington et al. (U.S. Publication No. 2021/0027112 A1), hereinafter Brewington, Eggers et al. (U.S. Publication No. 2009/0102858 A1), hereinafter Eggers, and Yen et al. (U.S. Publication No. 2019/0034734 A1), hereinafter Yen.
Regarding claim 3, Chen, Brewington, and Eggers teach the method as recited in claim 1,
wherein said first image analysis is performed on said image with said first resolution using a first machine learning model (Chen teaches that “an image may be first processed by a CNN to generate an initial segmentation” in para. [0055]. Here, this image is equivalent to the said image with the said first resolution),
wherein said second image analysis is performed on said segment of said image with said second resolution using a second machine learning model (Chen teaches that “the system may concatenate the initial segmentation with an up-sampled segmentation generated from a lower resolution and the input image for refinement. The concatenated matrix may be passed into another CNN to generate a final result” in para. [0055]. The “another” CNN is interpreted as equivalent to the claimed second machine learning model),
While Chen teaches “inside this second CNN, the filters may be weighted by the attention generated based on lower resolution features” in para. [0055], Chen, Brewington, and Eggers fail to teach wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes.
However, Yen teaches wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes (Yen teaches that “the deep learning network used by the DL-2 Forensic process 812 can use a larger network with more layers than the network of the DL-1 process”, wherein “the second deep learning network applied by the DL-2 Forensics process 812 can include more hidden layers than the deep learning network applied by the DL-1 process” as shown in para. [0130]. Yen teaches the hidden layers having nodes in para. [0137]).
Chen, Brewington, Eggers, and Yen are all considered to be analogous to the claimed invention because they are in the same field of using machine learning and resolution to carry out object detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington and Eggers) to incorporate the teachings of Yen and include “wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes”. The motivation for doing so would have been that, “the additional hidden layers allow the deep learning network of the DL-2 Forensics process 812 to determine more features of an object than the deep network of the DL-1 process can determine, leading to a more accurate classification of the object”, as suggested by Yen in para. [0130]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen, Brewington, and Eggers with Yen to obtain the invention specified in claim 3.
Regarding claim 10, Chen, Brewington, and Eggers teach the system as recited in claim 8,
wherein said first image analysis is performed on said image with said first resolution using a first machine learning model (Chen teaches that “an image may be first processed by a CNN to generate an initial segmentation” in para. [0055]. Here, this image is equivalent to the said image with the said first resolution),
wherein said second image analysis is performed on said segment of said image with said second resolution using a second machine learning model (Chen teaches that “the system may concatenate the initial segmentation with an up-sampled segmentation generated from a lower resolution and the input image for refinement. The concatenated matrix may be passed into another CNN to generate a final result” in para. [0055]. The “another” CNN is interpreted as equivalent to the claimed second machine learning model),
While Chen teaches “inside this second CNN, the filters may be weighted by the attention generated based on lower resolution features” in para. [0055], Chen, Brewington, and Eggers fail to teach wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes.
However, Yen teaches wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes (Yen teaches that “the deep learning network used by the DL-2 Forensic process 812 can use a larger network with more layers than the network of the DL-1 process”, wherein “the second deep learning network applied by the DL-2 Forensics process 812 can include more hidden layers than the deep learning network applied by the DL-1 process” as shown in para. [0130]. Yen teaches the hidden layers having nodes in para. [0137]).
Chen, Brewington, Eggers and Yen are all considered to be analogous to the claimed invention because they are in the same field of using machine learning and resolution to carry out object detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington and Eggers) to incorporate the teachings of Yen and include “wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes”. The motivation for doing so would have been that, “the additional hidden layers allow the deep learning network of the DL-2 Forensics process 812 to determine more features of an object than the deep network of the DL-1 process can determine, leading to a more accurate classification of the object”, as suggested by Yen in para. [0130]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen and Brewington with Yen to obtain the invention specified in claim 10.
Regarding claim 17, Chen, Brewington, and Eggers teach the system as recited in claim 15,
wherein said first image analysis is performed on said image with said first resolution using a first machine learning model (Chen teaches that “an image may be first processed by a CNN to generate an initial segmentation” in para. [0055]. Here, this image is equivalent to the said image with the said first resolution),
wherein said second image analysis is performed on said segment of said image with said second resolution using a second machine learning model (Chen teaches that “the system may concatenate the initial segmentation with an up-sampled segmentation generated from a lower resolution and the input image for refinement. The concatenated matrix may be passed into another CNN to generate a final result” in para. [0055]. The “another” CNN is interpreted as equivalent to the claimed second machine learning model),
While Chen teaches “inside this second CNN, the filters may be weighted by the attention generated based on lower resolution features” in para. [0055], Chen, Brewington, and Eggers fail to teach wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes.
However, Yen teaches wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes (Yen teaches that “the deep learning network used by the DL-2 Forensic process 812 can use a larger network with more layers than the network of the DL-1 process”, wherein “the second deep learning network applied by the DL-2 Forensics process 812 can include more hidden layers than the deep learning network applied by the DL-1 process” as shown in para. [0130]. Yen teaches the hidden layers having nodes in para. [0137]).
Chen, Brewington, Eggers, and Yen are all considered to be analogous to the claimed invention because they are in the same field of using machine learning and resolution to carry out object detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington and Eggers) to incorporate the teachings of Yen and include “wherein said second machine learning model is more complex than said first machine learning model by having a greater number of hidden layers of nodes”. The motivation for doing so would have been that, “the additional hidden layers allow the deep learning network of the DL-2 Forensics process 812 to determine more features of an object than the deep network of the DL-1 process can determine, leading to a more accurate classification of the object”, as suggested by Yen in para. [0130]. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen, Brewington, and Eggers with Yen to obtain the invention specified in claim 17.
Claims 6, 7, 13, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (U.S. Publication No. 2021/0133956 A1) in view of Brewington et al. (U.S. Publication No. 2021/0027112 A1), hereinafter Brewington, Eggers et al. (U.S. Publication No. 2009/0102858 A1), hereinafter Eggers, and Xiang (CN 110766644 A, see attached English translation for citations).
Regarding claim 6, Chen, Brewington, and Eggers teach the method as recited in claim 1, further comprising:
storing said images of areas of interest with multiple resolutions, wherein said image of said area of interest with said first resolution is retrieved from said stored images of areas of interest with multiple resolutions (Brewington teaches obtaining and storing sensor data including images of areas of interest at lower resolutions, and using one or more of these low resolution images to then determine an uncertainty level in order to determine whether to recapture images at a higher resolution as shown in para. [0020]-[0021]. Here, this process of utilizing the stored images at lower resolutions can be combined with Xiang’s teaching (below) of downsampling images of areas of interest with multiple resolutions to teach the above limitation).
Chen, Brewington, and Eggers fail to teach downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions.
However, Xiang teaches downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions (Xiang teaches “the specific numerical value of the first down-sampling multiple may be determined according to an instruction of a user”, wherein the process of down-sampling includes “determin[ing] several areas to be detected from the images to be detected according to a detection result, and then determine different first down-sampling multiples for each area to be detected according to different resolutions of targets in the different areas to be detected” in para. [0181]. See also para. [0158] wherein there exists at least a second image).
Chen, Brewington, Eggers, and Xiang are all considered to be analogous to the claimed invention because they are in the same field of using machine learning and resolution to carry out object detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington and Eggers) to incorporate the teachings of Xiang and include “downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions”. The motivation for doing so would have been that, “the detection rate of target detection is improved” and “the amount of computation [is reduced]”, as suggested by Xiang in para. [0168] and para. [0004], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen, Brewington, and Eggers with Xiang to obtain the invention specified in claim 6.
Regarding claim 7, Chen, Brewington, Eggers and Xiang teach the method as recited in claim 6,
wherein said second resolution has a higher resolution than said first resolution (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” in para. [0048] and FIG. 4 #440).
Regarding claim 13, Chen, Brewington, and Eggers teach the computer program product as recited in claim 8, wherein the program code further comprises the programming instructions for:
storing said images of areas of interest with multiple resolutions, wherein said image of said area of interest with said first resolution is retrieved from said stored images of areas of interest with multiple resolutions (Brewington teaches obtaining and storing sensor data including images of areas of interest at lower resolutions, and using one or more of these low resolution images to then determine an uncertainty level in order to determine whether to recapture images at a higher resolution as shown in para. [0020]-[0021]. Here, this process of utilizing the stored images at lower resolutions can be combined with Xiang’s teaching (below) of downsampling images of areas of interest with multiple resolutions to teach the above limitation).
Chen, Brewington, and Eggers fail to teach downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions.
However, Xiang teaches downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions (Xiang teaches “the specific numerical value of the first down-sampling multiple may be determined according to an instruction of a user”, wherein the process of down-sampling includes “determin[ing] several areas to be detected from the images to be detected according to a detection result, and then determine different first down-sampling multiples for each area to be detected according to different resolutions of targets in the different areas to be detected” in para. [0181]. See also para. [0158] wherein there exists at least a second image).
Chen, Brewington, Eggers, and Xiang are all considered to be analogous to the claimed invention because they are in the same field of using machine learning and resolution to carry out object detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington and Eggers) to incorporate the teachings of Xiang and include “downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions”. The motivation for doing so would have been that, “the detection rate of target detection is improved” and “the amount of computation [is reduced]”, as suggested by Xiang in para. [0168] and para. [0004], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen, Brewington, and Eggers with Xiang to obtain the invention specified in claim 13.
Regarding claim 14, Chen, Brewington, Eggers, and Xiang teach the method as recited in claim 13,
wherein said second resolution has a higher resolution than said first resolution (Chen teaches “performing a second image analysis of the set of uncertain regions at a second resolution of the tissue image that is greater than the first resolution” in para. [0048] and FIG. 4 #440).
Regarding claim 20, Chen, Brewington, and Eggers teach the system as recited in claim 15, wherein the program instructions of the computer program further comprise:
storing said images of areas of interest with multiple resolutions, wherein said image of said area of interest with said first resolution is retrieved from said stored images of areas of interest with multiple resolutions (Brewington teaches obtaining and storing sensor data including images of areas of interest at lower resolutions, and using one or more of these low resolution images to then determine an uncertainty level in order to determine whether to recapture images at a higher resolution as shown in para. [0020]-[0021]. Here, this process of utilizing the stored images at lower resolutions can be combined with Xiang’s teaching (below) of downsampling images of areas of interest with multiple resolutions to teach the above limitation).
Chen, Brewington, and Eggers fail to teach downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions.
However, Xiang teaches downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions (Xiang teaches “the specific numerical value of the first down-sampling multiple may be determined according to an instruction of a user”, wherein the process of down-sampling includes “determin[ing] several areas to be detected from the images to be detected according to a detection result, and then determine different first down-sampling multiples for each area to be detected according to different resolutions of targets in the different areas to be detected” in para. [0181]. See also para. [0158] wherein there exists at least a second image).
Chen, Brewington, Eggers, and Xiang are all considered to be analogous to the claimed invention because they are in the same field of using machine learning and resolution to carry out object detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Chen (as modified by Brewington and Eggers) to incorporate the teachings of Xiang and include “downsampling images of areas of interest with an original resolution with user-defined scaling factors forming images of areas of interest with multiple resolutions”. The motivation for doing so would have been that, “the detection rate of target detection is improved” and “the amount of computation [is reduced]”, as suggested by Xiang in para. [0168] and para. [0004], respectively. Therefore, it would have been obvious to one of ordinary skill at the time the invention was filed to combine Chen, Brewington, and Eggers with Xiang to obtain the invention specified in claim 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Phillips (U.S. Publication No. 2024/0328815 A1) teaches first and second modalities and merging said second image analysis result into said first image analysis result by replacing, in said first image analysis result, a portion corresponding to said area with said second image analysis result.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office
action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the
extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from
the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH
shortened statutory period, then the shortened statutory period will expire on the date the advisory
action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing
date of the advisory action. In no event, however, will the statutory period for reply expire later than
SIX MONTHS from the date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to KYLA G ALLEN whose telephone number is (703)756-5315. The examiner can
normally be reached M-F 7:30am - 4:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
John Villecco can be reached on (571) 272-7319. The fax phone number for the organization where this
application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/Kyla Guan-Ping Tiao Allen/
Examiner, Art Unit 2661
/JOHN VILLECCO/Supervisory Patent Examiner, Art Unit 2661
Prosecution Insights