Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in Sweden on 12/22/2023. It is noted, however, that applicant has not filed a certified copy of the SE2351501-8 application as required by 37 CFR 1.55.
Claim Objections
Claims 1, 5, 9 and 12 are objected to because of the following informalities:
In claim 1, Line 16, the word “or” appears to be missing after “... images, “ and before “an image generated ...”. Since the claim language states “based on a first input image comprising at least one of:” the word “or” should be included to make it easier to read and should then read as, “... based on a first input image comprising at least one of: the first depth map image, an image of the first set of images[[,]] or an image generated from a 3D-model of the region of interest at the first time period, ...”.
Additionally, in claim 1, Line 18 the word “or” appears to be missing after “... images, “ and before “... an image generated ...”. Since the claim language states “based on a second input image comprising at least one of:” the word “or” should be included to make it easier to read and should then read as, “and a second input image comprising at least one of: an image of the second set of images[[,]] or an image generated from a 3D-model of the region of interest at the second time period, ...”.
In claim 5, Line 6, the word “or” appears to be missing after “... period, “ and before “an image generated ...”. Since the claim language states “based on a first input image comprising at least one of:” the word “or” should be included to make it easier to read and should then read as, “... the first input image comprising at least one of: a first depth map image of the region of interest when observed at said attitude, position and first time period, an image of the region of interest when observed at said attitude, position and first time period, or an image generated from a 3D-model of the region of interest at the first time period, ...”
Additionally, in claim 5, Line 12, the word “or” appears to be missing after “... period, “ and before “an image generated ...”. Since the claim language states “the second input image comprising at least one of:” the word “or” should be included to make it easier to read and should then read as, “... the second input image comprising at least one of: an image of the region of interest when observed at said attitude, position and second time period, or an image generated from a 3D-model of the region of interest at the second time period, ...”.
In claim 9, Line 9, the word “or” appears to be missing after “flooding” and before “identification”. Since the claim language states “wherein the characteristics of a zone comprise at least one of:” the word “or” should be included to make it easier to read and should then read as, “wherein the characteristics of a zone comprise at least one of: identification of a zone comprising an infrastructure project, identification of a zone with a completed or non-completed building construction and/or building tear-down, identification of a zone being an agricultural field from identification of seasonal changes, identification of a zone of deforestation, identification of a zone comprising melting ice, such as a glacier, identification of a zone comprising a land slide, identification of a zone affected by an earth quake, identification of a zone affected by fire, identification of a zone affected by flooding, or identification of a zone comprising at least one vehicle.
In claim 12, Line 7, the word “or” appears to be missing after “... period, “ and before “an image generated ...”. Since the claim language states “based on a first input image comprising at least one of:” the word “or” should be included to make it easier to read and should then read as, “... the first input image comprising at least one of: a first depth map image of the region of interest when observed at said attitude, position and first time period, an image of the region of interest when observed at said attitude, position and first time period, or an image generated from a 3D-model of the region of interest at the first time period, ...”
Additionally, in claim 12, Line 13, the word “or” appears to be missing after “... period, “ and before “an image generated ...”. Since the claim language states “the second input image comprising at least one of:” the word “or” should be included to make it easier to read and should then read as, “... the second input image comprising at least one of: an image of the region of interest when observed at said attitude, position and second time period, or an image generated from a 3D-model of the region of interest at the second time period, ...”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5-7 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Long et al. (U.S. Patent: #12,061,253 B2) hereinafter Long, in view of Kawamura (Pub. No.: US 2015/0138322 A1).
Regarding claim 5, Long discloses a method for generating a relative depth map image (Col. 1, Lines 37-44 teaches that the systems and methods described herein include generation of a dense depth map using data from a camera and a radar on board a vehicle. For the purposes of this document, a “depth map” is defined as an image, i.e., a set of image data, that contains information relating to the distance of surfaces of scene objects from a viewpoint, typically by specifying distances of surfaces represented by pixels from the viewpoint.), the method comprising:
obtaining a first input image relating to a region of interest observed at a first time period (Col. 7, Lines 35-45 teach that the first trained network takes inputs from the camera data and the radar data. The inputs to the first trained network include the radar pixels 116, the image frame 118, optical flow, and radar flow. The optical flow describes a scene shift, i.e., differences in pixel coordinates corresponding to respective points in space, between the image frame 118 taken as input, which will be referred to as a first image frame 118, and a second image frame 118. The second image frame 118 can be from a different, nearby point in time from the first image frame 118. The second image frame 118 can be from before or after the first image frame 118.). However, Long fails to teach and at a specific attitude and position relative to the region of interest.
Kawamura discloses and at a specific attitude and position relative to the region of interest (Paragraph 39 teaches that the image processing unit 100 includes a first input unit 103 and a second input unit 104 to which the estimation result of the position and attitude change is input, a static region determining unit 105, a third input unit 106 relating to the estimation result of the past position and attitude change, and an entire position and attitude change estimation determining unit 107 and paragraph 81 teaches that in the present embodiment, a position and attitude change is estimated among a plurality of frames continuous along the time-axis direction, and the reference frame and the non-reference frame are associated with each other in the time-axis direction. The reference frame and the non-reference frame may not necessarily be adjacent frames. Additionally, paragraph 6 teaches that as a method for calculating a region of interest which is used for calculating a position and attitude change, a background region is often calculated. In the conventional background region extraction, a method for specifying a background and a moving object using a difference between continuous frames is typically employed.). Since Long teaches a method for receiving input images taken at different times for use in generating depth maps and Kawamura teaches a method for receiving input images at different time periods for use related to depth data and can determine an image’s position and attitude, it would have been obvious to a person having ordinary skill in the art to combine the features together so that any input images received for the use of generating a depth map, would also have the capabilities to include receiving position and attitude data related to those images.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long to incorporate the teachings of Kawamura, so that the combined features together would provide more accurate input images to be received for generating a depth map by incorporating attitude and position information related to the inputted image.
Furthermore, Long in view of Kawamura disclose the first input image comprising at least one of:
a first depth map image of the region of interest when observed at said attitude, position and first time period, an image of the region of interest when observed at said attitude, position and first time period, an image generated from a 3D-model of the region of interest at the first time period, the image generated from the 3D-model being generated at said attitude and position with respect to the region of interest (Paragraph 51 of Kawamura teaches that in step S401, image data obtained by the image acquiring device 101 and depth image data obtained by the depth image acquiring device 102 are input to the first processing unit 200. The first processing unit 200 calculates the first computation result and then outputs it to the first input unit 103 and paragraph 46 of Kawamura teaches that the processing result is output to the first estimating unit 205. The first estimating unit 205 executes processing for estimating a position and attitude change for each depth.);
obtaining a second input image relating to the region of interest observed at a second time period and at the corresponding attitude and position relative to the region of interest as for the first input image (Paragraph 41 of Kawamura teaches that the second input unit 104 to which the estimation result of the position and attitude change with use of a motion vector is input is connected to the second processing unit 300. A second position and attitude change computation result (hereinafter referred to as "second computation result") which has been estimated by the second processing unit 300 using a motion vector is output from the second input unit 104 to the static region determining unit 105 and the estimation determining unit 107.), the second input image comprising at least one of:
an image of the region of interest when observed at said attitude, position and second time period, an image generated from a 3D-model of the region of interest at the second time period, the image generated from the 3D-model being generated at said attitude and position with respect to the region of interest (Paragraph 52 of Kawamura teaches that in step S402, image data obtained by the image acquiring device 101 is input to the second processing unit 300. The second processing unit 300 calculates the second computation result and then outputs it to the second input unit 104. Additionally, paragraph 43 of Kawamura teaches that the estimation determining unit 107 executes second determination processing for determining a position and attitude change of the whole image using the first computation result and the second computation result and information about the static region determined by the static region determining unit 105. The position and attitude change computation result determined by the second determination processing is sent to the storage unit 108 for storage.);
and generating, using a neural network, the relative depth map image, based on the obtained first and second input images (Col. 8, Lines 32-36 of Long teach that the first trained network can be any suitable type of network for converting the inputs to the outputted confidence scores A(i, j, k). For example, the first trained network can be a convolutional neural network, which is well suited to analyzing visual imagery and Col. 10, Lines 23-30 of Long teach that with reference to FIG. 5, the computer 102 outputs a depth map 126 of projected depths for the respective camera pixels 120. The depth map 126 can include a depth for each camera pixel 120. The computer 102 outputs the depth map 126 based on the confidence scores. For example, the computer 102 outputs the depth map 126 based on the depth images 124, which are generated based on the confidence scores as described above.).
Regarding claim 6, Long in view of Kawamura disclose everything claimed as applied above (see claim 5), in addition, Long in view of Kawamura disclose generating a second depth map image based on the first depth map image and the generated relative depth image (Col. 11, Lines 21-23 of Long teach that next, in a block 630, the computer 102 outputs the depth map 126 by executing the second trained network with the depth images 124 as inputs, as described above.).
Regarding claim 7, Long in view of Kawamura disclose everything claimed as applied above (see claim 5), in addition, Long in view of Kawamura disclose detecting if the generated relative depth image meets a predetermined criterion relating to a measure of change of relative depth (Paragraph 130 of Kawamura teaches that in this case, the processing for setting the judgment criteria for the static region is executed by comparing all the estimation results of the position and attitude changes with use of depth data, all the estimation results of the position and attitude changes with use of a motion vector, with the estimation result of the past position and attitude change.).
Regarding claim 12, the system steps correspond to and are rejected similarly to the method steps of claim 5. In addition, Long in view of Kawamura discloses a system for generating a relative depth map image (Col. 1, Lines 37-44 of Long teaches that the systems and methods described herein include generation of a dense depth map using data from a camera and a radar on board a vehicle. For the purposes of this document, a “depth map” is defined as an image, i.e., a set of image data, that contains information relating to the distance of surfaces of scene objects from a viewpoint, typically by specifying distances of surfaces represented by pixels from the viewpoint.),
the system comprising processing circuitry (Col. 3, Lines 41-43 of Long teaches that the computer 102 is a microprocessor-based computing device, e.g., a generic computing device including a processor and a memory).
Regarding claim 13, Long in view of Kawamura disclose everything claimed as applied above (see claim 12), in addition, Long in view of Kawamura disclose wherein the processing circuitry comprises a processor and a memory, wherein the memory comprises instructions executable by said processor (Col. 3, Lines 54-60 of Long teaches that the computer 102 can thus include a processor, a memory, etc. The memory of the computer 102 can include media for storing instructions executable by the processor as well as for electronically storing data and/or databases, and/or the computer 102 can include structures such as the foregoing by which programming is provided.).
Claims 8, 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Long in view of Kawamura as applied to claim 5 above, and further in view of Wei et al. (Pub. No.: US 2016/0012633 A1), hereinafter Wei.
Regarding claim 8, Long in view of Kawamura disclose everything claimed as applied above (see claim 5), however, Long in view of Kawamura fail to disclose determining a volumetric change based on a plurality of generated relative depth images of the region of interest at different respective attitudes and positions with respect to the region of interest.
Wei discloses determining a volumetric change based on a plurality of generated relative depth images of the region of interest at different respective attitudes and positions with respect to the region of interest (Paragraph 45 teaches that more particularly, after depth map alignment and outlier identification, a three-dimensional model can be generated based at least in part on the plurality of depth maps. In particular, as an example, a volumetric fusion technique can be performed to merge the plurality of depth maps. For example, the volumetric fusion technique can average a plurality of signed distance functions respectively associated with the plurality of depth maps to generate a unified signed distance function for a volume enclosing the scene.) Since Long in view of Kawamura teach a method for generating depth maps and using depth images and data to determine information related to the attitude and position of regions within images and Wei teaches the volumetric fusion function of determining a volumetric change in depth images, it would have been obvious to a person having ordinary skill in the art to combine the features together so that additionally, a volumetric change related to the attitude and position of different regions and images could also be determined.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura to incorporate the teachings of Wei, so that the combined features together would improve the generation of the depth maps and provide better accuracy for reconstructing anything related to a 3-D model or depth image.
Regarding claim 10, Long in view of Kawamura disclose everything claimed as applied above (see claim 5), however, Long in view of Kawamura fail to disclose determining if a predetermined trigger level for triggering reconstruction of a 3D-model of the region of interest is exceeded based on at least one generated relative depth image.
Wei discloses determining if a predetermined trigger level for triggering reconstruction of a 3D-model of the region of interest is exceeded based on at least one generated relative depth image (Paragraph 151 teaches that outlier identification module 1418 can be implemented to identify one or more outlying points. For example, outlier identification module 1418 can be implemented to perform a pointwise outlier identification process to identify outlying points. In some embodiments, a confidence score associated with each outlying point can be reduced by a certain percentage. The confidence scores can be employed by additional modules, including, for example, depth map merging module 1420. Additionally, paragraphs 119-121 teach that the resulting depth maps can contain inaccurate, outlying regions, such as region 1102. Therefore, the outlier identification techniques of the present disclosure can be performed to identify such inaccurate regions by verifying the agreement of each depth map point with its nearby points from other depth maps. Once outliers are identified, their contribution to the final mesh can be downgraded or otherwise reduced or eliminated, resulting in more accurate three-dimensional surface reconstructions. As an example, as shown in simplified representation 1150 of FIG. 11B, the outlying region 1152 has been removed, resulting in a cleaner model surface.). Since Long in view of Kawamura teach a method for generating depth maps and using depth images and data to determine information related to the attitude and position of regions within images and Wei teaches a function for use in depth maps that allows for an outlier to be recognized and then perform a 3D reconstruction to improve the 3D model based on the triggering outlier, it would have been obvious to a person having ordinary skill in the art to combine the features together so that the 3D reconstruction triggering function could be used when generating depth maps related to a 3D scene or model.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura to incorporate the teachings of Wei, so that the combined features together would help improve any related 3D model reconstruction ability by removing outliers that don’t belong, which would then result in more accurate three-dimensional surface reconstructions.
Regarding claim 11, Long in view of Kawamura disclose everything claimed as applied above (see claim 5), however, Long in view of Kawamura fail to disclose wherein the first and/or second input image comprises satellite images and/or panchromatic images and/or synthetic aperture radar, SAR, images and/or aerial images.
Wei discloses wherein the first and/or second input image comprises satellite images and/or panchromatic images and/or synthetic aperture radar, SAR, images and/or aerial images (Paragraph 52 teaches that the images can be any suitable form of images, including, for example, satellite imagery, aerial imagery collected from an aircraft (e.g. as illustrated by aircraft icon 102), user-uploaded photographs, other imagery, or combinations thereof. Some images may include metadata indicating a pose or location of image capture while other images may not.). Since Long in view of Kawamura teach a method for generating depth maps and using a capturing device to capture depth images and data to determine information related to the attitude and position of regions within images and Wei teaches a method for using a capturing device to capture satellite images for generating depth maps, it would have been obvious to a person having ordinary skill in the art to combine the features together so that additional images, such as satellite images, could also be captured and utilized for depth map generation.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura to incorporate the teachings of Wei, so that the combined features together would provide additional ways to capture images related to depth map generation, including satellite images.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Long in view of Kawamura as applied to claim 5 above, and further in view of Pennings et al. (Pub. No.: US 2020/0258296 A1), hereinafter Pennings.
Regarding claim 9, Long in view of Kawamura disclose everything claimed as applied above (see claim 5), however, Long in view of Kawamura fail to disclose determining characteristics of zones in the generated relative depth image.
Pennings discloses determining characteristics of zones in the generated relative depth image (Paragraph 84 teaches that to determine the portions that extend past the edges of the scene in both the range and cross-range (or azimuth) dimensions, or the overhang, in some embodiments vendor metadata may be used to determine the location of one or more primary ambiguity zones. In some embodiments, these zones are determined with the sampling frequency in each dimension and the Nyquist relationship in distance from a given point or origin. The zones along the range dimension may be determined by the pulse repetition frequency of the scene. Azimuth zones may be determined by the location of the object relative to the flight path of the sensor and differential doppler frequency range that it can create before wrapping around to the other side of the image.). Since Long in view of Kawamura teach a method for generating depth maps and using a capturing device to capture depth images and data to determine information related to the attitude and position of regions within images and Pennings teaches using a capturing device to capture images for depth data usage and can determine different types of zones within the image, it would have been obvious to a person having ordinary skill in the art to combine the features together so that in addition to be able to recognize different regions of interest, different zones with different characteristics could also be recognized as well.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura to incorporate the teachings of Pennings, so that the combined features together would provide more accurate determinations of what is actually located in each region of interest or zone of a depth image.
Furthermore, Long in view of Kawamura and Pennings disclose wherein the characteristics of a zone comprise at least one of:
identification of a zone comprising an infrastructure project, identification of a zone with a completed or non-completed building construction and/or building tear-down, identification of a zone being an agricultural field from identification of seasonal changes, identification of a zone of deforestation, identification of a zone comprising melting ice, such as a glacier, identification of a zone comprising a land slide, identification of a zone affected by an earth quake, identification of a zone affected by fire, identification of a zone affected by flooding, identification of a zone comprising at least one vehicle (Paragraph 106 of Pennings teaches that through the use of a land-use land-cover (LULC) mask, the SAR collection can be filtered such that the change detection algorithm only searches within certain regions of the scene. In some embodiments, the LULC mask is user specified, determined automatically by analyzing the SAR data, or obtained through other suitable means. For example, the initial LULC mask can be downloaded from a United States Geological Survey (USGS) database, determined through Landsat 8, Sentinel-2, or Moderate Resolution Imaging Spectroradiometer (MODIS) satellite image data or another source, and/or determined through analysis. This localizes monitoring only to relevant areas, such as urban landscapes where construction/destruction of infrastructure is probable, further reducing falsely detected change and increasing detection precision.).
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Long in view of Kawamura and further in view of Chapdelaine-Couture et al. (Pub. No.: US 2024/0404179 A1), hereinafter C-C.
Regarding claim 1, Long discloses a method for training a neural network to generate a relative depth map image (Col. 1, Lines 37-44 teaches that the systems and methods described herein include generation of a dense depth map using data from a camera and a radar on board a vehicle. For the purposes of this document, a “depth map” is defined as an image, i.e., a set of image data, that contains information relating to the distance of surfaces of scene objects from a viewpoint, typically by specifying distances of surfaces represented by pixels from the viewpoint. Additionally, Col. 2, Lines 19-23 teaches that outputting the depth map may be based on the depth images. Outputting the depth map may include executing a trained network, and inputs to the trained network may include the radar pixels, the image frame, and the depth images.), the method comprising:
obtaining a first set of images relating to a region of interest within a first time period (Col. 7, Lines 35-45 teaches that the first trained network takes inputs from the camera data and the radar data. The inputs to the first trained network include the radar pixels 116, the image frame 118, optical flow, and radar flow. The optical flow describes a scene shift, i.e., differences in pixel coordinates corresponding to respective points in space, between the image frame 118 taken as input, which will be referred to as a first image frame 118, and a second image frame 118. The second image frame 118 can be from a different, nearby point in time from the first image frame 118. The second image frame 118 can be from before or after the first image frame 118.);
obtaining a second set of images relating to the region of interest within a second time period (Col. 10, Lines 33-36 teaches that the second trained network takes inputs from the camera data, the radar data, and the first trained network. The inputs to the second trained network include the radar pixels 116, the image frame 118, and the depth images 124.);
generating a first depth map image based on the first set of images (Col. 10, Lines 23-30 teaches that with reference to FIG. 5, the computer 102 outputs a depth map 126 of projected depths for the respective camera pixels 120. The depth map 126 can include a depth for each camera pixel 120. The computer 102 outputs the depth map 126 based on the confidence scores. For example, the computer 102 outputs the depth map 126 based on the depth images 124, which are generated based on the confidence scores as described above.). However, Long fails to disclose the first depth map image relating to a depth of the region of interest when observed at a certain attitude and position with respect to the region of interest.
Kawamura discloses the first depth map image relating to a depth of the region of interest when observed at a certain attitude and position with respect to the region of interest (Paragraph 39 teaches that the image processing unit 100 includes a first input unit 103 and a second input unit 104 to which the estimation result of the position and attitude change is input, a static region determining unit 105, a third input unit 106 relating to the estimation result of the past position and attitude change, and an entire position and attitude change estimation determining unit 107. Additionally, paragraph 49 teaches that the estimating unit 306 selects one of the position and attitude changes in the regions for each motion estimated by the position and attitude change estimating unit 304 and sets the selected one as the second computation result. For example, the position and attitude change of the region of which the proportion to the whole image is the greatest from among the regions for each motion is selected.). Since Long teaches a method for receiving input images taken at different times for use in generating depth maps and Kawamura teaches a method for receiving input images at different time periods for use related to depth data and can determine an image’s position and attitude, it would have been obvious to a person having ordinary skill in the art to combine the features together so that any input images received for the use of generating a depth map, would also have the capabilities to include receiving position and attitude data related to those images.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long to incorporate the teachings of Kawamura, so that the combined features together would provide more accurate input images to be received for generating a depth map by incorporating attitude and position information related to the inputted image.
However, Long in view of Kawamura fail to disclose generating a second depth map image based on the second set of images.
C-C discloses generating a second depth map image based on the second set of images (Paragraph 120 teaches that the method 1000 continues, in block 1030, with the device generating a second depth map by aligning one or more portions of the first depth map based on a control signal associated with the image of the physical environment. In some implementations, the second depth map is a derivative of the first depth map with at least some updated and/or different depth map values.). Since Long in view of Kawamura teach a method for generating depth maps and C-C teaches a method for generating multiple depth maps related to one another, it would have been obvious to a person having ordinary skill in the art to combine the features together so that multiple depth maps could be generated if required.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura to incorporate the teachings of C-C, so that the combined features together would allow for multiple depth maps to be able to be generated and that they could be used in relation to each other for any potential comparison or improvement.
Furthermore, Long in view of Kawamura and C-C disclose the second depth map image relating to a depth of the region of interest when observed at the corresponding attitude and position with respect to the region of interest as the first depth map image (Paragraph 143 of Kawamura teaches that the first position and attitude change estimating unit 2005 calculates a position and attitude change corresponding to the region calculated by the most compatible region calculating unit 2004, and then estimates the position and attitude change as the position and attitude change of the region.);
generating a first relative depth map image based on a difference between the first depth map image and the second depth map image (Paragraph 144 of C-C teaches that the method 1100 continues, in block 1140, with the device generating a second depth map by refining the one or more portions of the first depth map. In some implementations, the second depth map is a derivative of the first depth map with at least some updated and/or different depth map values. According to some implementations, at least some of the depth map values for the one or more portions of the first depth map are updated or upscaled when refining the one or more portions of the first depth map. Additionally, Col. 7, Lines 38-55 of Long teaches that the optical flow describes a scene shift, i.e., differences in pixel coordinates corresponding to respective points in space, between the image frame 118 taken as input, which will be referred to as a first image frame 118, and a second image frame 118. The second image frame 118 can be from a different, nearby point in time from the first image frame 118. The second image frame 118 can be from before or after the first image frame 118. For example, for a target observed in the first and second image frames 118, the optical flow gives a mapping from the pixel coordinates in the earlier of the two image frames 118 to the pixel coordinates in the later of the two image frames 118, e.g., Flow((i.sub.1,j.sub.1)).fwdarw.(i.sub.2,j.sub.2). The optical flow can be determined using known image-processing techniques for scene flow for images. Similarly, the radar flow describes shifting between the radar data at two points in time and can be determined using known techniques for scene flow for radar data.);
generating, using the neural network, a second relative depth map image, based on a first input image (Col. 2, Lines 49-50 of Long teaches that outputting the depth map may include executing a trained network and Col. 10, Lines 37-42 of Long teaches that the second trained network outputs the depth map 126, i.e., performs depth completion. The depth map 126 includes a depth for each camera pixel 120, meaning that the depth map 126 is dense. The depths are not limited to the depths of the radar pixels 116. The depths can blend along surfaces of objects that extend toward or away from the vehicle 100. Additionally, paragraph 71 of C-C teaches that in some implementations, the depth map refinement engine 842 corresponds to a guided filter, a joint bilateral filter, a machine learning module, or the like. For example, the machine learning module corresponds to a neural network, convolutional neural network, deep neural network, recurrent neural network, state vector machine, or the like. ) comprising at least one of:
the first depth map image, an image of the first set of images, an image generated from a 3D-model of the region of interest at the first time period (Paragraph 118 of C-C teaches that in some implementations, obtaining the first depth map includes obtaining a three-dimensional model of the physical environment and generating the first depth map, based on the three-dimensional model, a including a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment.), and a second input image comprising at least one of:
an image of the second set of images, an image generated from a 3D-model of the region of interest at the second time period (Paragraph 60 of C-C teaches that thus, in various implementations, the three-dimensional model is based on historical data obtained at one or more times before the unprocessed image 803 of the physical environment 105 was captured. For example, the three-dimensional model may be based on data obtained a second, a few seconds, a minute, a few minutes, an hour, a day, or any other time before the unprocessed image 803 of the physical environment 105 was captured.), wherein the first input image and the second input image have the corresponding attitude and position with respect to the region of interest (Paragraph 123 of Kawamura teaches that in both FIGS. 12A and 12B, the dynamic region occupies more area in an image than the static region. On the other hand, as shown in FIGS. 12C and 12D, the static region has greater proportion occupied by depth than the dynamic region. When the position and attitude change in the image shown in FIGS. 12A and 12B is estimated by the conventional method, a large number of motion vectors or corresponding points used for calculation appears in the dynamic region as compared with the static region. Thus, it is highly probable that the position of the dynamic region does not change as shown in FIG. 12F, resulting in readily obtaining the estimation result affected by the dynamic region.);
and changing parameters of the neural network based on a difference between the second relative depth map image and the first relative depth map image (Paragraph 144 of C-C teaches that in some implementations, the second depth map is a derivative of the first depth map with at least some updated and/or different depth map values. According to some implementations, at least some of the depth map values for the one or more portions of the first depth map are updated or upscaled when refining the one or more portions of the first depth map. In various implementations, the one or more portions of the first depth map are refined using a guided filter, a joint bilateral filter, a machine learning module, or the like. Additionally, paragraph 50 of C-C teaches that FIG. 7A illustrates a first transformed first image 701 generated by transforming the first image 502 based on the first depth map of the first image 502 and the difference between the left scene camera perspective and the left eye perspective.).
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Long in view of Kawamura and C-C as applied to claim 1 above, and further in view of Wei.
Regarding claim 2, Long in view of Kawamura and C-C disclose everything claimed as applied above (see claim 1), however, Long in view of Kawamura and C-C fail to disclose generating a first 3D-model of the region of interest at the first time period based on the first set of images, and wherein generating the first depth map image is further based on the first 3D-model.
Wei discloses generating a first 3D-model of the region of interest at the first time period based on the first set of images, and wherein generating the first depth map image is further based on the first 3D-model (Paragraph 145 teaches that pose determination module 1410 can be implemented to determine a pose for each of a plurality of images. For example, the pose for each image can describe a location and orientation in three-dimensional space at which such image was captured. Pose determination module 1410 can obtain the pose for each image from the device of capture (e.g. if the camera or other image capture device had accurate knowledge of its pose at the time of capture) or can derive or otherwise improve the pose for each image through an analysis of the plurality of images. Additionally, paragraphs 119 and 120 teach that the resulting depth maps can contain inaccurate, outlying regions, such as region 1102. Therefore, the outlier identification techniques of the present disclosure can be performed to identify such inaccurate regions by verifying the agreement of each depth map point with its nearby points from other depth maps. Lastly, paragraph 79 teaches that as an example, FIG. 6 depicts example depth maps according to an example embodiment of the present disclosure. In particular, graphical representation 600 depicts three-dimensional depth map volumes 602, 604, and 606 overlaid upon a three-dimensional model generated based on such depth maps.). Since Long in view of Kawamura and C-C teach methods for generating depth maps at different times and can incorporate data related to 3D depth models and Wei teaches a method for determining 3D models and scenes related to generating depth maps based on different times of when an image was captured and can generate 3D models based on related depth map information, it would have been obvious to a person having ordinary skill in the art to combine the features together so that 3D models could be generated for assistance towards generating and improving depth maps related to that 3D model.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura and C-C to incorporate the teachings of Wei, so that the combined features together would allow for the generation of 3D models related to generated depth maps and would improve the accuracy of generating depth maps corresponding to a 3D model and/or scene.
Regarding claim 3, Long in view of Kawamura and C-C disclose everything claimed as applied above (see claim 1), however, Long in view of Kawamura and C-C fail to disclose generating a second 3D-model of the region of interest at the second time period based on the second set of images, and wherein generating the second depth map image is further based on the second 3D-model.
Wei discloses generating a second 3D-model of the region of interest at the second time period based on the second set of images, and wherein generating the second depth map image is further based on the second 3D-model (Paragraph 123 teaches that referring again FIG. 2, after depth map alignment at (208) and outlier identification at (210), a three-dimensional model can be generated based at least in part on the plurality of depth maps. In particular, at (212) a volumetric fusion technique can be performed to merge the plurality of depth maps. For example, the volumetric fusion technique performed at (212) can average a plurality of signed distance functions respectively associated with the plurality of depth maps to generate a unified signed distance function for a volume enclosing the scene.). Since Long in view of Kawamura and C-C teach methods for generating depth maps at different times and can incorporate data related to 3D depth models and Wei teaches a method for determining different 3D models and scenes related to generating different depth maps based on different times of when an images were captured and can generate 3D models based on related depth map information, it would have been obvious to a person having ordinary skill in the art to combine the features together so that multiple 3D models could be generated for assistance towards generating and improving depth maps related to multiple 3D models.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura and C-C to incorporate the teachings of Wei, so that the combined features together would allow for the generation of multiple 3D models related to generated depth maps and would improve the accuracy of generating depth maps corresponding to a specific 3D model and/or scene.
Regarding claim 4, Long in view of Kawamura and C-C disclose everything claimed as applied above (see claim 1), however, Long in view of Kawamura and C-C fail to disclose wherein the first and second sets of images relating to the region of interest comprise satellite images and/or panchromatic images and/or synthetic aperture radar, SAR, images and/or aerial images.
Wei discloses wherein the first and second sets of images relating to the region of interest comprise satellite images and/or panchromatic images and/or synthetic aperture radar, SAR, images and/or aerial images (Paragraph 52 teaches that the images can be any suitable form of images, including, for example, satellite imagery, aerial imagery collected from an aircraft (e.g. as illustrated by aircraft icon 102), user-uploaded photographs, other imagery, or combinations thereof. Some images may include metadata indicating a pose or location of image capture while other images may not.). Since Long in view of Kawamura and C-C teach a method for generating depth maps and using a capturing device to capture depth images and data to determine information related to the attitude and position of regions within images and Wei teaches a method for using a capturing device to capture satellite images for generating depth maps, it would have been obvious to a person having ordinary skill in the art to combine the features together so that additional images, such as satellite images, could also be captured and utilized for depth map generation.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Long in view of Kawamura and C-C to incorporate the teachings of Wei, so that the combined features together would provide additional ways to capture images related to depth map generation, including satellite images.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sudry et al. (Pub. No.: US 2022/0198709 A1) teaches determining positions within a building site and constructing a 3D model of a building using depth maps
Gao et al. (Pub. No.: US 2023/0072702 A1) teaches training a neural network to generate depth maps based on speckle images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to George Renze whose telephone number is (703)756-5811. The examiner can normally be reached Monday-Friday 9:00am - 6:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.R./Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights