DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4 December 2025 has been entered.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 6 October 2025, 8 December 2025, and 6 January 2026 were received and the information disclosure statements have been considered by the examiner.
Response to Arguments
Claims 3, 4, 11, 13, and 16 have been amended. Claims 1-20 are pending in this action.
Applicant’s amendments of claims 3 and 13 with respect to the rejections of claims 3 and 13 under 35 U.S.C. 112(b) have been fully considered and are persuasive. Specifically, the applicant amended claims 3 and 13 to avoid the relative terminology previously present in the claims. The rejection of claims 3 and 13 under 35 U.S.C. 112(b) has been withdrawn.
Applicant’s arguments, see pg. 10-12, filed 4 December 2025, with respect to the rejection of claim 1 and claims dependent on claim 1 under 35 U.S.C. 103 have been fully considered and are not persuasive.
The applicant argues that as the examiner does not teach "projected mask" with Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) and that it is inappropriate to use Lim to teach a specific projected mask "that weights the one or more pixels based at least on proximity of the ego-object to the one or more detected objects". The examiner disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). The “projected mask” of claim 1 is taught by a combination of Lim in view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson), therefore, the applicant’s argument against Lim teaching the disclosed limitation is not persuasive because it is taught by a combination of references.
The applicant argues that Lim does not disclose weighting pixels of objects based on proximity but rather that "Lim is weighting its candidate combination lines - not detected object pixels in a projected mask", see applicant's remarks pg. 12. The examiner disagrees. The claim language states "updating a candidate position for a seam to an updated position based at least on an intersection of the seam at the candidate position with one or more pixels of one or more detected objects in at least one projected mask . . . that weights the one or more pixels based at least on proximity of the ego- object to the one or more detected objects;" emphasis added. By the broadest reasonable interpretation, the claim detects at least one object and weighs at least one pixel of that object. Lim discloses updating priority information, which is understood as weighting as it increases the likelihood of weighted or prioritized pixels being avoided by the combination line, based on the depth information of objects on the combination line (Lim, [0083] the seam is moved based on priority information, [0087] priority information includes depth or proximity, [0091] an example of how different priority information between candidate seams determines the location of the seam, see also [0076]-[0077] for information on the depth or proximity data). If the combination line is interpreted as a one pixel wide line, then at least one pixel of width through the height of the object is weighted based on proximity which reads on the “one or more pixels” recited in the claim. The arguments by the applicant appear to presume an interpretation in which the projected mask weighs each of the pixels of one or more detected objects, see "weighting detected object pixels based at least on proximity" applicant's remarks pg. 12. This is more narrow than the claim language. Therefore, the applicant's argument is not persuasive.
The applicant argues that Petterson does not disclose a projected mask that weighs one or more pixels of an object based on proximity. The examiner disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). The projected mask that weighs one or more pixels of an object based on proximity of claim 1 is taught by a combination of Lim in view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson), Lim discloses weighing one or more pixels based on proximity (Lim [0083], [0087], and [0091] as applied above) and Petterson discloses a projected mask (Petterson [0058] probability masks corresponding to the images being stitched together are projected onto a virtual image canvas. This is understood as at least one projected mask). The examiner understands a mask as an image with simplified pixel values to capture characteristics of the image, such as replacing pixel values of an object with a single color value or providing probabilities that pixels are associated with a query of interest. When considered in combination, it would be obvious to combine the projected mask of Petterson with the weighted pixels of Lim in order to compare the pixels of two images based on objects in the intersection of the images (Petterson, [0058]) and to minimize the difference in the intersection to generate a combination based on object weighting (Petterson [0051] objects with high-cost are understood to have a different weighting than objects with low cost as the cost or weighting changes how pixels associated with objects are considered). Therefore, the applicant’s argument against Petterson teaching the disclosed limitation is not persuasive because it is taught by a combination of references.
Therefore, the applicant’s arguments against the rejection of claim 1 and the claims dependent on claim 1 under 35 U.S.C. 103 are not persuasive and the rejection is maintained. Claims 11 and 16 likewise do not overcome the prior art in view of the applicant’s arguments responded to above.
Applicant’s arguments, see pg. 13, filed 4 December 2025, with respect to the rejection of claim 11 under 35 U.S.C. 103 have been fully considered and are not persuasive. The applicant argues that claim 11 has been amended to incorporate the allowable subject matter from claim 8. The examiner disagrees. In the action filed on 7 October 2025, the examiner stated that the allowable subject matter of claim 8 was "that the weighted saliency masks are weighted corresponding to binary object masks based at least on the proximity to the one or more detected objects", see the action filed on 7 October 2025 pg. 31. Claim 11 has not been amended to include any mention of binary object masks and therefore has not been amended to include the allowable subject matter of claim 8. Therefore, the applicant's argument is not persuasive.
Applicant’s arguments, see 13, filed 4 December 2025, with respect to the rejection of claim 16 under 35 U.S.C. 103 have been fully considered and are persuasive. Specifically, the applicant argues that Lim in view of Petterson does not disclose the amended language of claim 16 concerning a “ground projection”. The examiner agrees. The applicant also argues that the amendment to claim 16 causes the claim to no longer be interpreted as invoking 35 U.S.C. 112(f). The examiner agrees. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground of rejection is made in view of Gyori et al. (US 20180244199 A1; hereafter, Gyori). Gyori discloses:
at least one ground projection of a mask ([0067] images are projected onto the ground. [0018] and [0057] the blending is performed in the intersecting region of images by a mask. As the images are projected onto the ground when they are blended, the mask is understood to also be projected onto the ground)
Therefore, claim 16 remains rejected under 35 U.S.C. 103. The complete rejection, including motivations to combine, is below in the section “Claim Rejections - 35 USC § 103”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 7, 10-12, and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) in view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson).
Regarding claim 1, Lim discloses:
A method comprising: generating, using sensor data captured during a first time slice ([0039] each of the plurality of cameras collects image data "captured in the same time" which is understood as a first time slice), two or more image frames representative of two or more at least partially overlapping viewpoints ([0042] the plurality of cameras capture images with an overlapped view by the adjacent cameras. As a plurality of cameras are capturing images, it is understood that two or more images are captured) around an ego-object in an environment ([0149] the cameras may be arranged in an electronic device which may be understood as an ego-object as it is at the center of the arrangement of cameras. The object necessarily must be in whatever environment the cameras are imaging);
updating a candidate position for a seam to an updated position ([0047] "The processor 160 may perform stitching by a combination line by default settings and may change a location of the combination line", the "combination line by default settings" is understood as the "candidate position" and changing the location of the combination line is understood as an updated position) based at least on an intersection of the seam at the candidate position with one or more pixels of one or more detected objects ([0047] "may change a location of the combination line when the combination and coordinates of an object are overlapped with each other.");
and that weights the one or more pixels based at least on proximity of the ego-object to the one or more detected objects ([0083] priority information, understood to give priority to a certain position, i.e. a weighting, is updated based on the depth information, i.e. proximity, of objects in the image at the combination location. [0087] priority information includes depth or proximity. [0091] an example of how different priority information between candidate seams determines the location of the seam. The examiner understand weighting pixels as putting emphasis on pixels based on a characteristic such that the weighted pixels are treated differently than non-weighted pixels. Therefore, priority information is understood as giving weight to pixels such that pixels with high priority or weight are avoided in the combination. See also [0076]-[0077] for information on the depth or proximity data);
and generating a composite image frame ([0048] a composite, "stitched", image is generated) based at least on stitching the two or more image frames using the updated position of the seam ([0043] stitching is performed along the seam or combination line).
Lim does not disclose expressly at least one projected mask that is aligned with a three-dimensional coordinate system for the environment.
Pettersson discloses:
in at least one projected mask ([0058] probability masks corresponding to the images being stitched together are projected onto a virtual image canvas. This is understood as at least one projected mask) that is aligned with a three-dimensional coordinate system for the environment ([0057] camera locations and directions are known and images from separate cameras can be aligned and registered "based on known locations, positions, and/or another parameter(s) of the first camera 1104a and the second camera 1104b." Therefore, as the camera location is known and the relative positions of images are known such that they may be registered by translation and rotation, it is understood that the images and/or masks are aligned in a coordinate system for the environment)
Lim and Pettersson are combinable because they are from the same field of endeavor of stitching images together along a seam (Lim, [0005]; Pettersson, [0003]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the projected mask of Petterson with the weighted pixels of Lim.
The motivation for doing so would have been to compare the pixels of two images based on objects in the intersection of the images (Petterson, [0058]) and to minimize the difference in the intersection to generate a combination based on object weighting (Petterson [0051], objects with high-cost are understood to have a different weighting than objects with low cost as the cost or weighting changes how pixels associated with objects are considered. The cost or weighting is minimized for a combination line based on object cost or weighting).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify Lim with the projected mask in a three-dimensional coordinate system of Pettersson.
The motivation for doing so would have been that the camera location, direction, and coordinate information are "useable for projecting an image acquired by the camera" (Pettersson, [0026]).
Therefore, it would have been obvious to combine Pettersson with Lim to obtain the invention as specified in claim 1.
Regarding claim 2, Lim in view of Pettersson discloses the subject matter of claim 1.
Lim further discloses:
determining the candidate position for the seam in an overlapping region of the two or more image frames ([0047] "The processor 160 may perform stitching by a combination line by default settings and may change a location of the combination line", the "combination line by default settings" is understood as the "candidate position". [0043] the combination line is in the overlapping region of the images);
updating the candidate position for the seam to the updated position ([0047] "The processor 160 may perform stitching by a combination line by default settings and may change a location of the combination line", changing the location of the combination line is understood as updating the candidate position to an updated position) based at least on the intersection of the seam at the candidate position with the one or more pixels of the one or more detected objects ([0047] "may change a location of the combination line when the combination and coordinates of an object are overlapped with each other.")
Lim does not disclose expressly generating two or more projected masks comprising at least partially overlapping representation of the detected objects depicted in the image frames and an updated position based at least on the at least one projected mask.
Pettersson discloses:
wherein the updating of the candidate position for the seam to the updated position comprises: generating two or more projected masks ([0056] for each image, i.e. two or more, a probability map is generated. [0058] the probability maps are projected onto their respective images, and are therefore understood as "projected masks") comprising at least partially overlapping representations of the one or more detected objects depicted in the two or more image frames ([0032] the probability maps contain a probability for each pixel belonging to a class of object, i.e. representations of the one or more detected objects. They correspond to respective images. [0031] the respective images overlap in a portion of the scene, including objects such as people. Therefore, the associate probability maps are at least partially overlapping and represent the detected objects);
based at least on the intersection of the seam at the candidate position with the one or more detected objects in the at least one projected mask of the two or more projected masks ([0059] a seam is generated such that it does not intersect recognized objects in the probability maps, i.e. the projected masks, based on the cost map. [0058] the cost map is generated based on the probability maps).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify Lim with the projected masks of Pettersson.
The motivation for doing so would have been that "the perceptibility of the seam may be reduced as compared to methods that do not consider a likelihood of the seam intersecting one or more classes of objects" (Pettersson, [0021]).
Therefore, it would have been obvious to combine Pettersson with Lim to obtain the invention as specified in claim 2.
Regarding claim 5, Lim in view of Pettersson discloses the subject of claim 1. Lim does not disclose expressly generating the projected mask based at least on projecting two or more corresponding weighted saliency masks onto an overlapping portion of the two or more image frames, the weighted saliency masks representing a measure of saliency of each pixel in a corresponding one of the two or more image frames.
Pettersson discloses:
generating the at least one projected mask based at least on projecting two or more corresponding weighted saliency masks (The examiner is understanding "saliency" as objects that a human observer may be inclined to pay special attention. [0032] probability maps are generated which state the probability that each pixel belongs to a given class. Classes may be given higher priority than others such as a person in image, i.e. an object that a human observer is prone to give attention to. Therefore, the probability maps may be understood as weighted saliency masks) onto an overlapping portion of the two or more image frames ([0058] the probability maps are projected onto the corresponding portions of the respective images. [0053] Cameras have overlapping fields of view, therefore, the images and probability maps overlap as well), the two or more weighted saliency masks representing a measure of saliency of each pixel in a corresponding one of the two or more image frames ([0032] The probability masks indicate for each pixel the probability of the pixel belonging to a class of objects and indicates high priority, i.e. salient, classes of objects).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify Lim with the projected masks of Pettersson.
The motivation for doing so would have been that "the perceptibility of the seam may be reduced as compared to methods that do not consider a likelihood of the seam intersecting one or more classes of objects" (Pettersson, [0021]).
Therefore, it would have been obvious to combine Pettersson with Lim to obtain the invention as specified in claim 5.
Regarding claim 7, Lim in view of Pettersson discloses the subject matter of claim 1. Lim does not disclose expressly generating the projected mask based on two or more projecting masks corresponding weighted saliency masks, prioritizing one or more classes of the one or more detected objects, onto an overlapping portion of the two or more image frames.
Pettersson discloses:
further comprising generating the at least one projected mask based at least on projecting two or more corresponding weighted saliency masks ([0032] probability maps are generated which state the probability that each pixel belongs to a given class. Classes may be given higher priority that others such as a person in image, i.e. an object that a human observer is prone to give attention to. Therefore, the probability maps may be understood as weighted saliency mask), prioritizing one or more classes of the one or more detected objects ([0032] certain classes of objects are given higher priority than others), onto an overlapping portion of the two or more image frames ([0058] the probability maps are projected onto the corresponding portions of the respective images. [0053] Cameras have overlapping fields of view, therefore, the images and probability maps overlap as well).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify Lim with the projected masks of Pettersson.
The motivation for doing so would have been that "the perceptibility of the seam may be reduced as compared to methods that do not consider a likelihood of the seam intersecting one or more classes of objects" (Pettersson, [0021]).
Therefore, it would have been obvious to combine Pettersson with Lim to obtain the invention as specified in claim 7.
Regarding claim 10, Lim in view of Pettersson discloses the subject matter of claim 1. Lim further discloses:
wherein the method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources ([0180] the system may be implemented at least partially using cloud computing resources).
Regarding claim 11, Lim discloses:
A processor ([0036] and Fig. 1, the device may comprise a processor 160) comprising:
one or more circuits ([0036] the device may further comprise "a printed circuit board (PCB)" which is understood as one or more circuits. The device performs the mapped functions) to:
obtain, using one or more image sensors of an ego-machine ([0042] the plurality of cameras capture images with an overlapped view with the adjacent cameras), image data of two or more image frames ([0038] each of a plurality of cameras collects image data, i.e. two or more image frames) corresponding to two or more separate viewpoints that share at least a portion of an area in an environment ([0042] the plurality of cameras capture images with an overlapped view with the adjacent cameras, i.e. separate viewpoints that share a portion of an area);
detect one or more objects depicted in the two or more image frames ([0045] an object is detected in the images);
determine a candidate position for a seam ([0047] "The processor 160 may perform stitching by a combination line by default settings and may change a location of the combination line", the "combination line by default settings" is understood as the "candidate position") in an aligned representation of the two or more image frames ([0043] the combination line is in the overlapping region of the images, understood as an aligned representation);
update the candidate position of the seam to an updated position ([0047] "The processor 160 may perform stitching by a combination line by default settings and may change a location of the combination line", changing the location of the combination line is understood as updating the candidate position to an updated position) based at least on an intersection of the seam at the candidate position with one or more pixels that at least partially depict the one or more objects ([0047] "may change a location of the combination line when the combination and coordinates of an object are overlapped with each other.") and that are weighted by saliency based at least on proximity of the ego-machine to the one or more objects ([0083] priority information, understood to give priority to a certain position, i.e. a weighting, is updated based on the depth information, i.e. proximity, of objects in the image at the combination location. See also [0076]-[0077] for information on the depth or proximity data. The examiner is understanding the term "saliency" to mean applying an importance to an object based on a characteristic of interest. Therefore, the weighting by priority information is understood as a weighting by saliency);
and generate a composite image ([0048] a composite, "stitched", image is generated) based at least on stitching the two or more image frames using the updated position of the seam ([0043] stitching is performed along the seam or combination line).
Lim does not disclose expressly that the images are aligned with a three-dimensional coordinate system for the environment.
Pettersson discloses:
that is aligned with a three-dimensional coordinate system for the environment ([0057] camera locations and directions are known and images from separate cameras can be aligned and registered "based on known locations, positions, and/or another parameter(s) of the first camera 1104a and the second camera 1104b." Therefore, as the camera location is known and the relative positions of images are known such that they may be registered by translation and rotation, it is understood that the images and/or masks are aligned in a coordinate system for the environment);
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to modify Lim with the three-dimensional coordinate system of Pettersson.
The motivation for doing so would have been that the camera location, direction, and coordinate information are "useable for projecting an image acquired by the camera" (Pettersson, [0026]).
Therefore, it would have been obvious to combine Pettersson with Lim to obtain the invention as specified in claim 11.
Regarding claim 12, Lim in view of Pettersson discloses the subject matter of claim 11. Lim further discloses:
the one or more circuits further to update the candidate position for the seam to the updated position based at least on reducing a number of the one or more pixels ([0070] when it is determined that the candidate combination line intersects the recognized object the line may be moved. This is understood as avoiding intersecting the object which reduces the number of pixels of the combination line that intersect the object) in the aligned representation of the two or more image frames ([0067] the combination is in overlapping regions which are understood as an aligned representation. Fig 3 and [0068], item 330 is an aligned representation of item 310 and 320) that at least partially depict the one or more objects ([0065] the system recognizes objects in each image. Fig. 3, see item 352 detected in both image 310 and image 320).
Regarding claim 14, Lim in view of Pettersson discloses the subject matter of claim 11. Lim does not disclose expressly determining whether the seam intersect with the objects based on comparing the candidate position of the seam with corresponding pixels in two or more projected masks representing positions of the one or more detected objects in the aligned representation.
Pettersson discloses:
the one or more circuits further to determine whether the seam at the candidate position intersects the one or more pixels in the aligned representation that at least partially depict the one or more objects based at least on comparing the candidate position of the seam with corresponding pixels in two or more projected masks ([0059] The position of the seam is determined by comparing the projected masks, i.e. probability masks, in a cost map) representing positions of the one or more detected objects ([0032] probability masks indicate the probability of a pixel belonging to an object class, including locations in an image) in the aligned representation ([0058] the probability maps are projected onto the corresponding portions of the respective images, i.e. overlapping regions, which is understood as an aligned representation).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lim with the projected masks of Pettersson.
The motivation for doing so would have been that "the perceptibility of the seam may be reduced as compared to methods that do not consider a likelihood of the seam intersecting one or more classes of objects" (Pettersson, [0021]).
Therefore, it would have been obvious to combine Pettersson with Lim to obtain the invention as specified in claim 14.
Regarding claim 15, Lim in view of Pettersson discloses the subject matter of claim 11. Lim further discloses:
wherein the processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources ([0180] the system may be implemented at least partially using cloud computing resources).
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) in view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson) in further view of Song et al. (CN 113905176 A; hereafter, Song).
Regarding claim 3, Lim in view of Pettersson discloses the subject matter of claim 1.
Lim in view of Petterson does not disclose that updating the candidate position for the seam cases the seam to switch between a lateral position and a front position in an aligned representation of the two or more image frames.
Song discloses:
The method of claim 1, wherein the updating of the candidate position for the seam to the updated position causes the seam to switch between a first position in a region lateral to the ego-object and a second position in a region in front of the ego-object ([0087] and [0089], the seam switches from a position lateral to the vehicle, see Fig. 5 position B1, to a position in front of the vehicle, see Fig. 5 position B3) in an aligned representation of the two or more image frames ([0067] several cameras are present and the seam is adjusted in an area, such as A1, where the field of view of the cameras overlap, which is understood as an aligned representation of the two or more frames).
Song is combinable with Lim in view of Pettersson because it is in the same field of endeavor of a panoramic image stitching method (Song, [0001]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the seam switching as taught by Song with the invention of Lim in view of Pettersson.
The motivation for doing so would have been "In the above examples, full consideration is given to the different emphasis on vision requirements in scenarios corresponding to different vehicle speeds. When the vehicle is driving slowly, the side of the vehicle can be fully displayed. When the vehicle is driving quickly, the horizontal field of view in front of the vehicle is fully expanded, effectively improving the driver's visual experience and ensuring driving safety" (Song, [0090]).
Therefore, it would have been obvious to combine Song with Lim in view of Petterson to obtain the invention as specified in claim 3.
Regarding claim 13, Lim in view of Pettersson discloses the subject matter of claim 11.
Lim in view of Petterson does not disclose that updating the candidate position for the seam cases the seam to switch between a lateral position and a front position in an aligned representation of the two or more image frames.
Song discloses:
The processor of claim 11, the one or more circuits further to update the candidate position for the seam to the updated position based at least on switching between a first position in a region lateral to the ego-machine and a second position in a region in front of the ego-machine ([0087] and [0089], the seam switches from a position lateral to the vehicle, see Fig. 5 position B1, to a position in front of the vehicle, see Fig. 5 position B3) in the aligned representation of the two or more image frames ([0067] several cameras are present and the seam is adjusted in an area, such as A1, where the field of view of the cameras overlap, which is understood as an aligned representation of the two or more frames).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the seam switching as taught by Song with the invention of Lim in view of Pettersson.
The motivation for doing so would have been "In the above examples, full consideration is given to the different emphasis on vision requirements in scenarios corresponding to different vehicle speeds. When the vehicle is driving slowly, the side of the vehicle can be fully displayed. When the vehicle is driving quickly, the horizontal field of view in front of the vehicle is fully expanded, effectively improving the driver's visual experience and ensuring driving safety" (Song, [0090]).
Therefore, it would have been obvious to combine Song with Lim in view of Petterson to obtain the invention as specified in claim 13.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) in view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson) in further view of Holzer et al. (U.S. Publ. No. 20170148223 A1; hereafter, Holzer).
Regarding claim 4, Lim in view of Pettersson discloses the subject matter of claim 1.
Lim further discloses:
and determining whether the candidate position for the seam intersects the one or more detected objects ([0047] "may change a location of the combination line when the combination and coordinates of an object are overlapped with each other.")
Lim in view of Pettersson does not disclose expressly projecting two or more object masks corresponding to two or more frames onto an overlapping portion of the two or more image frames.
Holzer discloses:
The method of claim 1, further comprising generating the at least one projected mask based at least on projecting two or more object masks corresponding to the two or more frames ([0131] the binary masks are used to "mask out pixels in every frame to extract the . . . object of interest out of the frames." The process of masking out is understood as projecting the masks onto images. As this is done for "every frame" it is understood to generate at least two or more object masks corresponding to the two or more frames) onto an overlapping portion of the two or more image frames ([0089] images include the object of interest , i.e. segmented or binary object, and are overlapping. Therefore, projecting the binary masks would be projecting on an overlapping portion),
in the two or more projected object masks in the overlapping portion of the two or more frames ([0131] the binary masks are generated from probability maps. [0130] the probability map indicates the pixels which are likely to correspond to the object(s) of interest).
Holzer is combinable with Lim in view of Pettersson because Holzer is in the same field of endeavor of generating surround view images (Holzer, [0045]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lim in view of Pettersson with the binary masks of Holzer.
The motivation for doing so would have been that binary masks provide a "frame that is sharply aligned to the boundaries and temporally consistent (non-fluctuating)" (Holzer, [0131]).
Therefore, it would have been obvious to combine Holzer with Lim in view of Pettersson to obtain the invention as specified in claim 4.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) in view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson) in further view of Khwaja et al. (U.S. Publ. No. 20200020075 A1; hereafter, Khwaja).
Regarding claim 6, Lim in view of Pettersson discloses the subject matter of claim 1. Lim in view of Pettersson does not discloses expressly generating the at least one projected mask based on omitting a subset of an initial set of detected objects determined to be beyond a threshold distance from the ego-object.
Khwaja discloses:
generating the at least one projected mask based at least on omitting a subset of an initial set of detected objects ([0073] objects near to the camera may be stitched separately from objects far from the camera. This may be understood as omitting an initial set of detected objects due to distance) determined to be beyond a threshold distance from the ego-object ([0073] a threshold distance may be used to determine which objects to consider).
Khwaja is combinable with Lim in view of Pettersson because Holzer is in the same field of endeavor of generating a surround view image or a panoramic image (Khwaja, [0035]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lim in view of Pettersson with the distance based mask of Kwhaja.
The motivation for doing so would have been that "a depth sensor may provide the ability to stretch, compress, or warp portions of an image of an object located close to camera system 210, resulting in an improved rendering of the object in a stitched image" (Khwaja, [0073]).
Therefore, it would have been obvious to combine Khwaja with Lim in view of Pettersson to obtain the invention as specified in claim 6.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) in view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson) in further view of Ratnakar (U.S. Publ. No. 20160006943 A1).
Regarding claim 9, Lim in view of Pettersson discloses the subject matter of claim 1. Lim in view of Pettersson does not disclose expressly a medical probe or a multi-view corresponding to the medical probe.
Ratnakar discloses:
wherein the ego-object is a medical probe ([0006] the device is an endoscope which is understood as a medical probe), and the two or more at least partially overlapping viewpoints comprise at least one of a multi-view or a proximity view corresponding to the medical probe ([0006] multiple cameras provide multiple views which are merged into a consolidated view which is understood as a multi-view).
Ratnakar is combinable with Lim in view of Pettersson because Ratnakar is in the same field of endeavor of generating a composite image of adjacent images (Ratnakar, [0042]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lim in view of Pettersson with the medical probe of Ratnakar.
The motivation for doing so would have been "greatly facilitating the operator of the endoscope (such as a physician doctor) how the distal end should be turned so that the discovered lesion or polyp falls within the “intervention zone”" (Ratnakar, [0046]), therefore making medical procedures with a medical probe easier.
Therefore, it would have been obvious to combine Ratnakar with Lim in view of Pettersson to obtain the invention as specified in claim 9.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) in view of Gyori et al. (US 20180244199 A1; hereafter, Gyori).
Regarding claim 16, Lim discloses:
A system comprising: one or more hardware processors ([0036] and Fig. 1, the device may comprise a processor 160) to update a candidate position for a seam ([0047] "The processor 160 may perform stitching by a combination line by default settings and may change a location of the combination line", changing the location of the combination line is understood as updating the candidate position to an updated position) in an overlapping region of two or more image frames ([0067] the combination is in overlapping regions. Fig 3 and [0068], item 330 is an aligned representation of item 310 and 320) based at least on an intersection with an initial candidate position of the seam and one or more pixels corresponding to one or more detected objects ([0047] "may change a location of the combination line when the combination and coordinates of an object are overlapped with each other.")
that weights the one or more pixels based at least on proximity of the ego-object to the one or more detected objects ([0083] priority information, understood to give priority to a certain position, i.e. a weighting, is updated based on the depth information, i.e. proximity, of objects in the image at the combination location. See also [0076]-[0077] for information on the depth or proximity data) and to generate a composite image ([0048] a composite, "stitched", image is generated) based at least on stitching the two or more image frames using the seam at the updated candidate position ([0043] stitching is performed along the seam or combination line).
Lim does not disclose expressly at least one ground projection of a mask.
Gyori discloses:
at least one ground projection of a mask ([0067] images are projected onto the ground. [0018] and [0057] the blending is performed in the intersecting region of images by a mask. As the images are projected onto the ground when they are blended, the mask is understood to also be projected onto the ground)
Gyori is combinable with Lim because it is from the same field of endeavor of dynamically placing a stitching line dependent on objects in stitched images (Gyori, [0012]).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention to combine the ground projection of a mask of Gyori with the invention of Lim.
The motivation for doing so would have been "The present invention provides the following advantages. The visibility of the surrounding objects in a bird's eye view by the surround view system is improved. In addition, the visibility of 3D objects in a seamless surround view is not limited in the stitched area, because the image synthesizing may not fade out the view of the 3D object. Dependent on the driving conditions, the environmental conditions of the vehicle and further input like the position of the 3D objects around the image synthesizer can provide a feedback for the image synthesizer to adapt the step of combining of images. Hence, dynamical changing of the image synthesizing improves the view of the vehicle surroundings" (Gyori, [0093]).
Further, the examiner understands “a mask” as a representation of an image that is simplified to emphasize certain characteristics, for example, replacing objects in an image with pixels of a single tone in order to better differentiate the objects and the rest of the image. As such, a mask is an image. Therefore, it would be obvious to perform the steps of claim 16 on a ground projection of a mask because doing so would be a simple substitution for one known element, the images of Lim, for another element, the projected ground masks of Gyori, to yield predictable results, dynamically placed stitching seams which avoid objects in the image.
Therefore, it would have been obvious to combine Gyori with Lim to obtain the invention as specified in claim 16.
Claims 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lim et al. (U.S. Publ. No. 20190272619 A1; hereafter, Lim) in view of Gyori et al. (US 20180244199 A1; hereafter, Gyori) in further view of Pettersson et al. (U.S. Publ. No. 20200265622 A1; hereafter, Pettersson).
Regarding claim 17, Lim in view of Gyori discloses the subject matter of claim 16.
Lim in view of Gyori does not disclose expressly generating two or more projected masks corresponding to detected objects base on projecting two or more corresponding weighted saliency masks onto an overlapping portion of two or more image frames.
Pettersson discloses:
wherein the one or more processing units update the candidate position for the seam by generating two or more projected masks ([0056] for each image, i.e. two or more, a probability map is generated. [0058] the probability maps are projected onto their respective images, and are therefore understood as "projected masks") corresponding to the one or more detected objects ([0028] objects are detected in classes. [0032] classes are considered in generating probability maps, i.e. projected masks) based at least on projecting two or more corresponding weighted saliency masks ([0032] probability maps are generated which state the probability that each pixel belongs to a given class. Classes may be given higher priority that others such as a person in image, i.e. an object that a human observer is prone to give attention to. Therefore, the probability maps may be understood as weighted saliency masks) onto an overlapping portion of the two or more image frames ([0058] the probability maps are projected onto the corresponding portions of the respective images. [0053] Cameras have overlapping fields of view, therefore, the images and probability maps overlap as well)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the projection masks of Pettersson with the invention of Lim in view of Gyori.
The motivation for doing so would have been that "the perceptibility of the seam may be reduced as compared to methods that do not consider a likelihood of the seam intersecting one or more classes of objects" (Pettersson, [0021]).
Therefore it would have been obvious to combine Pettersson with Lim in view of Gyori to obtain the invention as specified in claim 17.
Regarding claim 18, Lim in view of Gyori in further view of Pettersson discloses the subject matter of claim 17.
Lim in view of Gyori does not disclose expressly that the weighted saliency masks represent a measure of saliency of at least one pixel in a corresponding one of the two or more image frames.
Pettersson discloses:
wherein the two or more weighted saliency masks represent a measure of saliency of at least one pixel ([0032] each pixel is considered for the probability of belonging to an object class and some object classes are given higher priority, which is understood as saliency) in a corresponding one of the two or more image frames ([0032] the pixels are considered in each image, which is understood as one of the two or more image frames).
Regarding claim 19, Lim in view of Gyori in further view of Pettersson discloses the subject matter of claim 17.
Lim in view of Gyori does not disclose expressly that the weighted saliency masks represent a measure of saliency of at least one pixel in a corresponding one of the two or more image frames.
Pettersson discloses:
wherein projecting the two or more corresponding weighted saliency masks onto an overlapping portion of the two or more image frames comprises prioritizing one or more object classes of the one or more detected objects ([0032] certain classes of objects are given higher priority than others, understood as prioritizing one or more object classes).
Regarding claim 20, Lim in view of Gyori in further view of Pettersson discloses the subject matter of claim 16.
Lim further discloses:
wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for generating synthetic data; or a system implemented at least partially using cloud computing resources ([0180] the system may be implemented at least partially using cloud computing resources).
Allowable Subject Matter
Claim 8 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 8, the closest prior art, Pettersson et al. (U.S. Publ. No. 20200265622; hereafter, Pettersson), discloses generating at least one projected mask based at least on projected two or more corresponding weighted saliency masks onto an overlapping potion of the two or more image frames. The prior art does not disclose or reasonably suggest that the weighted saliency masks are weighted corresponding to binary object masks based at least on the proximity to the one or more detected objects. The prior art did not teach both binary object masks and weighting the binary object masks based on the distance of detected objects.
The claim as a whole is found non-obvious over the prior art including:
weighting corresponding binary object masks based at least on proximity to the one or more detected objects,
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20170372147 A1, Stervik et al., discloses a system of stitching bird’s eye view images which are understood as ground projection images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA B CROCKETT whose telephone number is (571)270-7989. The examiner can normally be reached Monday-Thursday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA B. CROCKETT/ /JOHN VILLECCO/Examiner, Art Unit 2661 Supervisory Patent Examiner, Art Unit 2661