DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/23/2026 has been entered.
Response to Arguments
Applicant’s arguments, see Remarks pages 12, filed 01/23/2026, with respect to the rejections of claim(s) 1, 4, 6-7, 14, and 16-17 under 35 U.S.C. 103 have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Objections
Claims 5 & 16 are objected to because of the following informalities:
Claim 5 limitation “andperform the object detection on the new transformed data…” should be corrected to “and perform the object detection on the new transformed data...”.
Claim 16 limitation “wherein the aggregating the results comprises merging object masks associated with each iteration to obtain a combined object mask” should be corrected to “wherein the aggregating of the results comprises merging object masks associated with each iteration to obtain a combined object mask”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 6-7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites the limitation "determine an incremental detection amount based at least on a comparison of the object mask with one or more existing object masks". There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the limitation is interpreted as "determine an incremental detection amount based at least on a comparison of the combined object mask with one or more existing object masks".
Claim 7 recites the limitation "determine an incremental object mask as the difference between the object mask and one or more existing object masks". There is insufficient antecedent basis for this limitation in the claim. For the purposes of examination, the limitation is interpreted as "determine an incremental object mask as a difference between the combined object mask and one or more existing object masks".
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4, 9, 13-14, 17, and 19 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhang et al. (US-20220044365-A1) hereinafter referenced as Zhang.
Regarding claim 1, Zhang discloses: A system comprising: one or more processors to execute a processing pipeline to detect an object based at least on iteratively transforming and processing input data that includes at least one representation of the object, and aggregating results of each of at least two iterations to generate a combined object detection result (Zhang: Abstract: “Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a plurality of neural networks in a multi-branch pipeline to generate image masks for digital images…the disclosed system can utilize separate neural networks to generate a first mask portion for a portion of the digital image including a defined boundary region and a second mask portion for a portion of the digital image including a blended boundary region. The disclosed system can generate the mask portion for the blended boundary region by utilizing a trimap generation neural network to automatically generate a trimap segmentation including the blended boundary region. The disclosed system can then merge the first mask portion and the second mask portion to generate an image mask for the digital image.”; Wherein the input image is iteratively processed in order to extract and merge blended boundary region and defined boundary region masks.),
wherein for a subsequent iteration following an initial iteration, the one or more processors are further to update at least one value of one or more data transformation parameters relative to a value used in the initial iteration and to generate new transformed data by applying a data transformation to original input data based on the at least one updated value (Zhang: 0075: “In one or more embodiments, the semantic cropping neural network 302 further crops the digital image 300 based on the boundary of the object. For instance, the semantic cropping neural network 302 detects at least one salient object in the digital image 300 using object detection. The semantic cropping neural network 302 then generates at least one cropped digital image 300 corresponding to one or more portions of the digital image 300 including the salient object(s). In some embodiments, the mask generation system 102 generates a separate cropped digital image for each portion of the digital image including an object.”;
0121: “the mask generation system 102 generates a final image mask 618 , illustrated in FIG. 6J, by joining the combined image mask with an image mask for the remaining portions of the digital image 600.”;
Wherein the cropping of the input image based on salient object detection prior to generating and merging first and second mask portions constitutes the generation of transformed data by applying a data transformation to original input data based on updated data transformation parameters.).
Regarding claim 4, Zhang discloses: The system of claim 1, wherein the one or more processors are to perform the iteratively transforming and processing until a determination that a termination criteria has been satisfied, and wherein the aggregating of the results comprises merging object masks associated with two or more iterations to obtain a combined object mask (Zhang: 0075: “the semantic cropping neural network 302 detects at least one salient object in the digital image 300 using object detection. The semantic cropping neural network 302 then generates at least one cropped digital image 300 corresponding to one or more portions of the digital image 300 including the salient object(s). In some embodiments, the mask generation system 102 generates a separate cropped digital image for each portion of the digital image including an object”;
0121: “Additionally, in one or more embodiments, the mask generation system 102 generates a final image mask 618, illustrated in FIG. 6J, by joining the combined image mask with an image mask for the remaining portions of the digital image 600 .”; Wherein the iteration until all salient objects images have been processed constitutes a termination criteria.).
Regarding claim 9, Zhang discloses: wherein the one or more processors are further to: select a data transformation type corresponding to the iteratively transforming based at least on an object type corresponding to the object (Zhang: 0072: “As previously mentioned, in response to classifying a digital image 300 as a portrait image (or a soft-object image), the mask generation system 102 utilizes a semantic cropping neural network 302 to crop one or more portions of the digital image 300 based an object in the digital image 300 . ”; 0075: “the semantic cropping neural network 302 detects at least one salient object in the digital image 300 using object detection. The semantic cropping neural network 302 then generates at least one cropped digital image 300 corresponding to one or more portions of the digital image 300 including the salient object(s). In some embodiments, the mask generation system 102 generates a separate cropped digital image for each portion of the digital image including an object”).
Regarding claim 13, Zhang discloses: The system of claim 1, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for presenting one or more of virtual reality content, augmented reality content, or mixed reality content; a system for hosting one or more real-time streaming applications; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Zhang: 0040: “As shown in FIG. 1, the server device(s) 104 includes or hosts the image editing system 110…In one or more embodiments, the image editing system 110 uses the digital images in a variety of applications such as databases of digital media assets, digital video presentations, digital advertisements, virtual or augmented reality environments, or other environments that utilize digital images (including digital video). In one or more embodiments, the image editing system 110 provides modified digital images to another system such as a system/application at the client device 106 or to a third-party system.”;
0161: “Embodiments of the present disclosure can also be implemented in cloud computing environments. ”).
As per claim(s) 14, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 1 are analogous.
Regarding claim 19, Zhang discloses: The processor of claim 17, wherein a number of processing iterations of the plurality of processing iterations is determined based at least on a termination criteria (Zhang: 0075: “the semantic cropping neural network 302 detects at least one salient object in the digital image 300 using object detection. The semantic cropping neural network 302 then generates at least one cropped digital image 300 corresponding to one or more portions of the digital image 300 including the salient object(s). In some embodiments, the mask generation system 102 generates a separate cropped digital image for each portion of the digital image including an object”;
0121: “Additionally, in one or more embodiments, the mask generation system 102 generates a final image mask 618, illustrated in FIG. 6J, by joining the combined image mask with an image mask for the remaining portions of the digital image 600 .”; Wherein the iteration number is determined based on the number of salient objects images to be processed.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 8, 15-16, 18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, and further in view of Wei et al. (Fusion of an Ensemble of Augmented Image Detectors for Robust Object Detection) hereinafter referenced as Wei.
Regarding claim 2, Zhang discloses: The system of claim 1.
Zhang does not disclose expressly: wherein for at least one iteration, the one or more processors are further to: perform object detection on the new transformed data according to one or more values of one or more object detection parameters to generate an object mask.
Wei discloses: a method for performing image-based object detection by fusing the detection results of a plurality of object detection algorithms (Wei: Abstract: “A significant challenge in object detection is accurate identification of an object’s position in image space, whereas one algorithm with one set of parameters is usually not enough, and the fusion of multiple algorithms and/or parameters can lead to more robust results. Herein, a new computational intelligence fusion approach based on the dynamic analysis of agreement among object detection outputs is proposed. Furthermore, we propose an online versus just in training image augmentation strategy.”). Wherein for at least one iteration of performing object detection on an input image, one or more processors are configured to: perform object detection on a new transformed input image according to one or more values of one or more object detection parameters to generate an object detection (Wei: Figure 2; 3.1. Overview: “For the proposed system, during the in-line (testing) phase, the input goes through three stages. First, the input is augmented to produce several variations, so we can have augmented inputs for future stages. Second, a detector is applied to obtain AABBs and related labels in each augmented image. Practically, this would be implemented by applying multiple detectors in parallel, one for each of the augmented images.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the system for generating object masks disclosed by Zhang by implementing the algorithms for image augmentation prior to performing object detection taught by Wei. The suggestion/motivation for doing so would have been “For augmentation of each input, the goal is to produce a range of inputs. The “optimal” augmentation cannot be determined algorithmically, and it varies depending on the type of objects detected, the image background, etc. Instead, a range of images of varying quality is generated and presented to the detector. The augmentation methods used herein include changing brightness, contrast, edge enhancement, global histogram equalization, Gaussian blurring and adding independent and identically distributed (IID) Gaussian noise to simulate different scenarios…we choose to focus on basic operations, which we have already shown lead to success.” (Wei: 3.2 Augmentation). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang with Wei to obtain the invention as specified in claim 2.
Regarding claim 8, Zhang discloses: The system of claim 1.
Zhang does not disclose expressly: wherein the iteratively transforming comprises applying a spatial filter to the input data.
Wei discloses: a method for performing image-based object detection by fusing the detection results of a plurality of object detection algorithms (Wei: Abstract: “A significant challenge in object detection is accurate identification of an object’s position in image space, whereas one algorithm with one set of parameters is usually not enough, and the fusion of multiple algorithms and/or parameters can lead to more robust results. Herein, a new computational intelligence fusion approach based on the dynamic analysis of agreement among object detection outputs is proposed. Furthermore, we propose an online versus just in training image augmentation strategy.”). The method comprises performing several iterations of object detection on an augmented input image, wherein the augmentations comprise applying a spatial filter to the input image (Wei: Figure 2; 4.2.1. Augmentation Methods: “In the experiments performed, the Python Imaging Library (PIL) is utilized [40] to obtain augmented images. In PIL brightness and contrast enhancement classes, a factor is used for the change of brightness and contrast. When this factor is one, it gives the original image. Based on the experimental evaluation, in ADAS examples, factor values are chosen to be 1, 0.25, 0.5, 1.5, 2.0 and 2.5 for brightness and contrast. To add Gaussian noise, the noise variance is chosen to be 0.001, 0.003 and 0.005. This is based on qualitative image assessment, since these noise levels do not drastically alter the appearance of the input image. Other augmentation methods, including edge enhancement, global histogram equalization and Gaussian blurring (radius = 2), are predefined image operations in PIL.”; Wherein augmentations, such as Gaussian blurring constitute spatial filters.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the system for generating object masks disclosed by Zhang by implementing the algorithms for image augmentation prior to performing object detection taught by Wei. The suggestion/motivation for doing so would have been “For augmentation of each input, the goal is to produce a range of inputs. The “optimal” augmentation cannot be determined algorithmically, and it varies depending on the type of objects detected, the image background, etc. Instead, a range of images of varying quality is generated and presented to the detector. The augmentation methods used herein include changing brightness, contrast, edge enhancement, global histogram equalization, Gaussian blurring and adding independent and identically distributed (IID) Gaussian noise to simulate different scenarios…we choose to focus on basic operations, which we have already shown lead to success.” (Wei: 3.2 Augmentation). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang with Wei to obtain the invention as specified in claim 8.
As per claim(s) 15, arguments made in rejecting claim(s) 2 are analogous.
Regarding claim 16, Zhang and Wei discloses: The method of claim 15, wherein the iteratively transforming and processing is performed until a determination that a termination criteria has been satisfied, and wherein the aggregating of the results comprises merging object masks associated with each iteration to obtain a combined object mask (Zhang: 0075: “the semantic cropping neural network 302 detects at least one salient object in the digital image 300 using object detection. The semantic cropping neural network 302 then generates at least one cropped digital image 300 corresponding to one or more portions of the digital image 300 including the salient object(s). In some embodiments, the mask generation system 102 generates a separate cropped digital image for each portion of the digital image including an object”;
0121: “Additionally, in one or more embodiments, the mask generation system 102 generates a final image mask 618, illustrated in FIG. 6J, by joining the combined image mask with an image mask for the remaining portions of the digital image 600 .”; Wherein the iteration until all salient objects images have been processed constitutes a termination criteria.).
As per claim(s) 18, arguments made in rejecting claim(s) 2 are analogous.
Regarding claim 20, Zhang discloses: The processor of claim 17.
Zhang does not disclose expressly: wherein the processor comprises parallel processing circuitry to accelerate performing the plurality of processing iterations with respect to the input data.
Wei discloses: the application of parallel processing detectors to perform object detection on transformed data in parallel (Wei: Figure 2; 3.1. Overview: “For the proposed system, during the in-line (testing) phase, the input goes through three stages. First, the input is augmented to produce several variations, so we can have augmented inputs for future stages. Second, a detector is applied to obtain AABBs and related labels in each augmented image. Practically, this would be implemented by applying multiple detectors in parallel, one for each of the augmented images…the system produces variations of each input, detects objects in each variation and fuses the top T results for each object, with the expectation of getting a more accurate AABB for each object in the input.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the multiple processing units performing object detection on multiple augmented inputs in parallel further taught by Wei in order to perform the object mask detection disclosed by Zhang in parallel. The suggestion/motivation for doing so would have been “Another thing that may slow down the system is there are multiple augmented inputs that need to be processed. We can avoid the slowdown either via multiple processing units or a more powerful processing unit.” (Wei: Section 4.3. Discussion). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang with Wei to obtain the invention as specified in claim 20.
Claim(s) 3 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, and further in view of Siriborvornratanakul et al. (Multiscale Visual Object Detection for Unsupervised Ubiquitous Projection Based on a Portable Projector-Camera System) hereinafter referenced as Siriborvornratanakul, and Wei.
Regarding claim 3, Zhang discloses: The system of claim 1.
Zhang does not disclose expressly: wherein the one or more processors comprise parallel processing circuitry to accelerate application of the data transformation and performance of the object detection.
Siriborvornratanakul discloses: wherein the one or more processors comprise parallel processing circuitry to accelerate application of the data transformation (Siriborvornratanakul: Figure 6; Section III. D. Parallel implementation: “Considering recent growth in multicore processors and parallel programming languages, we changed the sequential implementation to the equivalent parallel implementation. The concept of our parallel implementation to the multiscale visual detection is illustrated in Fig. 6. The preprocessed image is distributed simultaneously to all scales. Instead of using the same smoothing parameters as the sequential implementation, the parallel implementation increases the smoothing effect by enlarging directly the value of σ G used in each scale.”; Wherein the input image is processed through the Gaussian smoothing filter sequentially.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the multicore processors, used to transform the input image in parallel, disclosed by Siriborvornratanakul, to perform the semantic image cropping disclosed by Zhang. The suggestion/motivation for doing so would have been “Our experiments presented in Fig. 6 show that the parallel implementation offers similar multiscale detection outcomes compared with the sequential implementation. In this way, speed of detection can be improved significantly with few modifications.” (Siriborvornratanakul: Section III. D. Parallel implementation). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Zhang in view of Siriborvornratanakul does not disclose expressly: wherein the one or more processors comprise parallel processing circuitry to accelerate…performance of the object detection.
Wei discloses: wherein the one or more parallel processing units are applied to perform object detection on transformed data in parallel (Wei: Figure 2; 3.1. Overview: “For the proposed system, during the in-line (testing) phase, the input goes through three stages. First, the input is augmented to produce several variations, so we can have augmented inputs for future stages. Second, a detector is applied to obtain AABBs and related labels in each augmented image. Practically, this would be implemented by applying multiple detectors in parallel, one for each of the augmented images…the system produces variations of each input, detects objects in each variation and fuses the top T results for each object, with the expectation of getting a more accurate AABB for each object in the input.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of performing object detection in parallel further disclosed by Wei by using the parallel units disclosed by Zhang in view of Siriborvornratanakul to perform the iterative object detection in parallel. The suggestion/motivation for doing so would have been “Another thing that may slow down the system is there are multiple augmented inputs that need to be processed. We can avoid the slowdown either via multiple processing units or a more powerful processing unit.” (Wei: Section 4.3. Discussion). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang in view of Siriborvornratanakul with Wei to obtain the invention as specified in claim 3.
Regarding claim 12, Zhang in view of Siriborvornratanakul discloses: The system of claim 11.
Zhang in view of Siriborvornratanakul does not disclose expressly: wherein the one or more parallel processing units are further to: perform the object detection on the new transformed data for multiple iterations in parallel.
Wei discloses: wherein the one or more parallel processing units are applied to perform object detection on transformed data for multiple iterations in parallel (Wei: Figure 2; 3.1. Overview: “For the proposed system, during the in-line (testing) phase, the input goes through three stages. First, the input is augmented to produce several variations, so we can have augmented inputs for future stages. Second, a detector is applied to obtain AABBs and related labels in each augmented image. Practically, this would be implemented by applying multiple detectors in parallel, one for each of the augmented images…the system produces variations of each input, detects objects in each variation and fuses the top T results for each object, with the expectation of getting a more accurate AABB for each object in the input.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of performing object detection in parallel further taught by Wei by using the parallel units disclosed by Zhang in view of Siriborvornratanakul to perform the iterative object detection in parallel. The suggestion/motivation for doing so would have been “Another thing that may slow down the system is there are multiple augmented inputs that need to be processed. We can avoid the slowdown either via multiple processing units or a more powerful processing unit.” (Wei: Section 4.3. Discussion). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang in view of Siriborvornratanakul with Wei to obtain the invention as specified in claim 12.
Claim(s) 5-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Wei, and further in view of Hu et al. (US2022028087A1) hereinafter referenced as Hu.
Regarding claim 5, Zhang in view of Wei discloses: The system of claim 2, wherein the one or more processors are further to: update at least one value of at least one of the one or more object detection parameters; and perform the object detection on the new transformed data according to the at least one updated value of the at least one of the one or more object detection parameters to obtain a new object mask.
Zhang in view of Wei does not disclose expressly: wherein the one or more processors are further to: responsive to a determination that a termination criteria has not been satisfied, update at least one value of at least one of the one or more object detection parameters.
Thus, Zhang in view of Wei does not disclose expressly: the calculation of a defined boundary region mask and a blended boundary region mask for an augmented input image based on a determined termination criteria.
Hu discloses: a method of determining a target region of an object within an image by iteratively updating an object mask (Hu: Abstract). Wherein, responsive to a determination that a termination criteria has not been satisfied, update the object mask (Hu: 0140-0143: “Step S104 : Determine, in a case that an updated original mask satisfies an error convergence condition, a target image region of the target object in the target image according to the updated original mask…the terminal device calculates an error between an original mask before the updating and an original mask after the updating, and detects whether the error is less than a preset error threshold. If the error is less than the preset error threshold, it indicates that the updated original mask satisfies the error convergence condition, and in this case, the terminal device may use the updated original mask as a target mask. If the error is not less than the preset error threshold, it indicates that the updated original mask does not satisfy the error convergence condition, and the original mask needs to be updated continuously and iteratively, until the updated original mask satisfies the error convergence condition.”)
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the error convergence condition taught by Hu into the method for generating object masks disclosed by Zhang in view of Wei by terminating the system based on an iterative mask difference. The suggestion/motivation for doing so would have been “the terminal device calculates an error between an original mask before the updating and an original mask after the updating, and detects whether the error is less than a preset error threshold. If the error is less than the preset error threshold, it indicates that the updated original mask satisfies the error convergence condition, and in this case, the terminal device may use the updated original mask as a target mask” (Hu: 0143; Wherein the thresholds for determining error convergence condition serve as a method for determining whether there is a “convergence” between iteration results such that an optimal mask result has been determined.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang in view of Wei with Hu to obtain the invention as specified in claim 5.
Regarding claim 6, Zhang discloses: The system of claim 4.
Zhang does not disclose expressly: wherein the one or more processors are further to: determine an incremental detection amount based at least on a comparison of the object mask with one or more existing object masks (Claim limitation is interpreted according to the claim interpretation in the rejection of claim 6 under 35 U.S.C. 112(b) disclosed above).
Thus, Zhang does not disclose expressly: a comparison between a combined object and one or more existing object masks in order to determine an incremental object detection amount.
Wei discloses: a method for performing image-based object detection by fusing the detection results of a plurality of object detection algorithms (Wei: Abstract: “A significant challenge in object detection is accurate identification of an object’s position in image space, whereas one algorithm with one set of parameters is usually not enough, and the fusion of multiple algorithms and/or parameters can lead to more robust results. Herein, a new computational intelligence fusion approach based on the dynamic analysis of agreement among object detection outputs is proposed. Furthermore, we propose an online versus just in training image augmentation strategy.”). Wherein one or more processors are configured to: perform at least one comparison between an object detection result with one or more existing object detection results (Wei: Figure 2; 3.1. Overview: “For the proposed system, during the in-line (testing) phase, the input goes through three stages. First, the input is augmented to produce several variations, so we can have augmented inputs for future stages. Second, a detector is applied to obtain AABBs and related labels in each augmented image. Practically, this would be implemented by applying multiple detectors in parallel, one for each of the augmented images… In summary, the system produces variations of each input, detects objects in each variation and fuses the top T results for each object, with the expectation of getting a more accurate AABB for each object in the input. The overview of the proposed system is shown in Figure 2, and Algorithm 1 is a formal description of the proposed system.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the system for generating object masks disclosed by Zhang by implementing the algorithms for image augmentation prior to performing object detection taught by Wei. The suggestion/motivation for doing so would have been “For augmentation of each input, the goal is to produce a range of inputs. The “optimal” augmentation cannot be determined algorithmically, and it varies depending on the type of objects detected, the image background, etc. Instead, a range of images of varying quality is generated and presented to the detector. The augmentation methods used herein include changing brightness, contrast, edge enhancement, global histogram equalization, Gaussian blurring and adding independent and identically distributed (IID) Gaussian noise to simulate different scenarios…we choose to focus on basic operations, which we have already shown lead to success.” (Wei: 3.2 Augmentation; Wherein the usage of multiple object detection results from various augmentations serve to improve overall detection results.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Zhang in view of Wei does not disclose expressly: wherein the one or more processors are further to: determine an incremental detection amount based at least on a comparison of the object mask with one or more existing object masks (Claim limitation is interpreted according to the claim interpretation in the rejection of claim 6 under 35 U.S.C. 112(b) disclosed above); and determine that the termination criteria has been satisfied based at least on the incremental detection amount.
Hu discloses: a method of determining a target region of an object within an image by iteratively updating an object mask (Hu: Abstract). Wherein one or more processors are configured to: determine an incremental detection amount based at least on a comparison of an object mask with one or more existing object masks; and determine that the termination criteria has been satisfied based at least on the incremental detection amount (Hu: 0140-0143: “Step S104 : Determine, in a case that an updated original mask satisfies an error convergence condition, a target image region of the target object in the target image according to the updated original mask…the terminal device calculates an error between an original mask before the updating and an original mask after the updating, and detects whether the error is less than a preset error threshold. If the error is less than the preset error threshold, it indicates that the updated original mask satisfies the error convergence condition, and in this case, the terminal device may use the updated original mask as a target mask. If the error is not less than the preset error threshold, it indicates that the updated original mask does not satisfy the error convergence condition, and the original mask needs to be updated continuously and iteratively, until the updated original mask satisfies the error convergence condition.”)
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the error convergence condition taught by Hu into the method for generating object masks disclosed by Zhang in view of Wei by terminating the system based on an iterative mask difference. The suggestion/motivation for doing so would have been “the terminal device calculates an error between an original mask before the updating and an original mask after the updating, and detects whether the error is less than a preset error threshold. If the error is less than the preset error threshold, it indicates that the updated original mask satisfies the error convergence condition, and in this case, the terminal device may use the updated original mask as a target mask” (Hu: 0143; Wherein the thresholds for determining error convergence condition serve as a method for determining whether there is a “convergence” between iteration results such that an optimal mask result has been determined.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang in view of Wei with Hu to obtain the invention as specified in claim 6.
Regarding claim 7, Zhang discloses: The system of claim 4.
Zhang does not disclose expressly: wherein the one or more processors are further to: determine an incremental object mask as the difference between the object mask and one or more existing object masks (Claim limitation is interpreted according to the claim interpretation in the rejection of claim 7 under 35 U.S.C. 112(b) disclosed above).
Thus, Zhang does not disclose expressly: the determination of a difference between a combined object mask and one or more existing object masks in order to determine an incremental object mask.
Wei discloses: a method for performing image based object detection by fusing the detection results of a plurality of object detection algorithms (Wei: Abstract: “A significant challenge in object detection is accurate identification of an object’s position in image space, whereas one algorithm with one set of parameters is usually not enough, and the fusion of multiple algorithms and/or parameters can lead to more robust results. Herein, a new computational intelligence fusion approach based on the dynamic analysis of agreement among object detection outputs is proposed. Furthermore, we propose an online versus just in training image augmentation strategy.”). Wherein one or more processors are configured to: perform at least one comparison between an object detection result with one or more existing object detection results (Wei: Figure 2; 3.1. Overview: “For the proposed system, during the in-line (testing) phase, the input goes through three stages. First, the input is augmented to produce several variations, so we can have augmented inputs for future stages. Second, a detector is applied to obtain AABBs and related labels in each augmented image. Practically, this would be implemented by applying multiple detectors in parallel, one for each of the augmented images… In summary, the system produces variations of each input, detects objects in each variation and fuses the top T results for each object, with the expectation of getting a more accurate AABB for each object in the input. The overview of the proposed system is shown in Figure 2, and Algorithm 1 is a formal description of the proposed system.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the system for generating object masks disclosed by Zhang by implementing the algorithms for image augmentation prior to performing object detection taught by Wei. The suggestion/motivation for doing so would have been “For augmentation of each input, the goal is to produce a range of inputs. The “optimal” augmentation cannot be determined algorithmically, and it varies depending on the type of objects detected, the image background, etc. Instead, a range of images of varying quality is generated and presented to the detector. The augmentation methods used herein include changing brightness, contrast, edge enhancement, global histogram equalization, Gaussian blurring and adding independent and identically distributed (IID) Gaussian noise to simulate different scenarios…we choose to focus on basic operations, which we have already shown lead to success.” (Wei: 3.2 Augmentation; Wherein the usage of multiple object detection results from various augmentations serve to improve overall detection results.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Zhang in view of Wei does not disclose expressly: wherein the one or more processors are further to: determine an incremental object mask as the difference between the object mask and one or more existing object masks (Claim limitation is interpreted according to the claim interpretation in the rejection of claim 7 under 35 U.S.C. 112(b) disclosed above).
Hu discloses: a method of determining a target region of an object within an image by iteratively updating an object mask (Hu: Abstract). Wherein one or more processors are configured to: determine an incremental object mask as the difference between the an object mask and one or more existing object masks (Hu: 0140-0143: “Step S104: Determine, in a case that an updated original mask satisfies an error convergence condition, a target image region of the target object in the target image according to the updated original mask…the terminal device calculates an error between an original mask before the updating and an original mask after the updating, and detects whether the error is less than a preset error threshold. If the error is less than the preset error threshold, it indicates that the updated original mask satisfies the error convergence condition, and in this case, the terminal device may use the updated original mask as a target mask.”;
0145-0149: “An original mask before the updating is mask1, an original mask after the updating is mask2, a target image is Is, and a first error l1 may be determined according to the following formula (4):
PNG
media_image1.png
72
307
media_image1.png
Greyscale
…a second error l2 may be determined according to the formula (5):
PNG
media_image2.png
66
204
media_image2.png
Greyscale
…The first error l1 and the second error l2 are added to obtain an error I between the original mask before the updating and the original mask after the updating:
PNG
media_image3.png
23
77
media_image3.png
Greyscale
”; Wherein the calculation of error between mask iterations based on mask subtraction, constitutes the determination of an incremental object mask).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the error convergence condition taught by Hu into the method for generating object masks disclosed by Zhang in view of Wei by terminating the system based on an iterative mask difference. The suggestion/motivation for doing so would have been “the terminal device calculates an error between an original mask before the updating and an original mask after the updating, and detects whether the error is less than a preset error threshold. If the error is less than the preset error threshold, it indicates that the updated original mask satisfies the error convergence condition, and in this case, the terminal device may use the updated original mask as a target mask” (Hu: 0143; Wherein the thresholds for determining error convergence condition serve as a method for determining whether there is a “convergence” between iteration results such that an optimal mask result has been determined.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang in view of Wei with Hu to obtain the invention as specified in claim 7.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang, and further in view of Von Berg et al. (US-20080205716-A1) hereinafter referenced as Von Berg.
Regarding claim 10, Zhang discloses: The system of claim 1.
Zhang does not disclose expressly: wherein the input data corresponds to a plurality of voxels and a result of the object detection is an object mask identifying at least a subset of the plurality of voxels corresponding to the object.
Von Berg discloses: wherein the input data corresponds to a plurality of voxels (Von Berg: 0003: “Any kind of multi-dimensional image data may be segmented according to the invention, including medical and non-medical image data, two-dimensional image data like common pictures, three-dimensional image data like volume image data or a temporal sequence of two-dimensional image data, four-dimensional image data like a temporal sequence of volume image data, and even higher-dimensional image data.”; Wherein the 3D image data corresponds to voxels) and a result of the object detection is an object mask identifying at least a subset of the plurality of voxels corresponding to the object (Von Berg: 0021: “once a model-based segmentation is completed and a model representing the image object is obtained, the model may be used for a coarse distinction between voxels of the object and voxels of the background. According to this distinction a threshold value, e.g. a gray value, may be chosen or calculated which can be used for the data-driven segmentation.”.
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the algorithms for segmenting the 3D image data based on model-based and data-driven segmentation algorithms disclosed by Von Berg into the system for generating object masks disclosed by Zhang. The suggestion/motivation for doing so would have been “It is an object of the present invention to provide an image processing device and a corresponding image processing method which overcome the drawbacks of the known methods described above and allow for a segmentation which is robust in terms of vulnerability to image artifacts and accurate in terms of reflecting anatomical details, wherein the results of the segmentation are achieved within a reasonable, short processing time.” (Von Berg: 0010: Wherein the process allows for robust, accurate, and quick 3D object segmentation). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang with Von Berg to obtain the invention as specified in claim 10.
Claim(s) 11 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang, and further in view of Siriborvornratanakul.
Regarding claim 11, Zhang discloses: The system of claim 1.
Zhang does not disclose expressly: wherein the one or more processors comprise one or more parallel processing units, and wherein the one or more parallel processing units apply the data transformation to the input data for multiple iterations in parallel.
Siriborvornratanakul discloses: wherein the one or more processors comprise one or more parallel processing units, and wherein the one or more parallel processing units apply the data transformation to the input data for multiple iterations in parallel (Siriborvornratanakul: Figure 6; Section III. D. Parallel implementation: “Considering recent growth in multicore processors and parallel programming languages, we changed the sequential implementation to the equivalent parallel implementation. The concept of our parallel implementation to the multiscale visual detection is illustrated in Fig. 6. The preprocessed image is distributed simultaneously to all scales. Instead of using the same smoothing parameters as the sequential implementation, the parallel implementation increases the smoothing effect by enlarging directly the value of σ G used in each scale.”; Wherein the input image is processed through the Gaussian smoothing filter sequentially.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the multicore processors, used to transform the input image in parallel, disclosed by Siriborvornratanakul, to perform the semantic image cropping disclosed by Zhang. The suggestion/motivation for doing so would have been “Our experiments presented in Fig. 6 show that the parallel implementation offers similar multiscale detection outcomes compared with the sequential implementation. In this way, speed of detection can be improved significantly with few modifications.” (Siriborvornratanakul: Section III. D. Parallel implementation). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Zhang with Siriborvornratanakul to obtain the invention as specified in claim 11.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights