DETAILED ACTION
Claims 1-9 are pending
Claims 1, 8 and 9 are independent
Claims 1, 8 and 9 are amended
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/17/2026 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Applicant's arguments filed 02/23/2026 have been fully considered but they are not persuasive. Applicant argues, see Applicant’s remarks pgs.6-7, that Examiner did not respond to the argument that stated that prior art Beck does not disclose that its transformation function generates a ‘distorted segmentation image.’
Examiner would like to point out, that although Beck does not specifically say, “a two-dimensional distortion”, Beck discusses a thin-plate spline transformation. The claim Beck is mapped to is claim 5, which states “wherein the two-dimensional distortion applied to the segmentation image includes a thin-plate spline transformation.”
Specifically:
(Beck, paragraph 0035, “The transformation functions are most preferably so-called 'thin-plate spline' transformations in each case.”)
Further, Tang also teaches a ‘distorted segmentation image’ as cited in claim 1, see Tang, pg. 133119, Col. 2, paragraph 2, “Therefore, we combined the statistical shape model and the 3D thin plate spline algorithm (3D-TPS) to simulate the generation of 3D medical images. 3D thin plate spline algorithm can give the deformation of the whole space between one real image and one simulated image via the corresponding position of N matching points in the two images.” And paragraph 1, “In addition, this medical image augmentation strategy is also applicable to 2D medical images.”
Therefore, the 35 USC 103 rejection is maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 and 6-9 are rejected under 35 U.S.C. 103 as being unpatentable over Tribelhorn et al (US Published Patent Application No. 20090175537, "Tribelhorn"), in view of Tang et al (An Augmentation Strategy for Medical Image Processing Based on Statistical Shape Model and 3D Thin Plate Spline for Deep Learning) and in further view of Planche et al (WO Published Patent Application No. 2019192744, "Planche").
In regard to claim 1 and analogous claims 8 and 9, Tribelhorn teaches generating an edge image using an edge detection applied to a digital image, the digital image including one or more digital objects, the edge image including a plurality of edge pixels determined as representing edges of the one or more digital objects in the digital image; (Tribelhorn, paragraph 0012, “In yet another aspect of the present patent document, a method for processing a digital image comprising an imaged document and surrounding image is provided, wherein the method comprises the steps of: finding potential edges of said imaged document by at least two different computer implemented edge detection techniques; grouping the found potential edges into top, bottom, left and right side potential edge groups [a plurality of edge pixels determined as representing edges of the one or more digital objects in the digital image;] ; for each edge group, selecting a subset of potential edges that are determined to likely represent an edge of the imaged document [generating an edge image using an edge detection applied to a digital image,]; generating a plurality of edge sets from the subsets of potential edges;”)
selecting edge-pixels from the plurality of edge pixels; (Tribelhorn, paragraph 0012, “…grouping the found potential edges into top, bottom, left and right side potential edge groups;”)
providing a segmentation image using the digital image, the segmentation image including one or more segments representing the one or more digital objects, wherein the segmentation image includes a plurality of first pixels, positions of the first pixels in the segmentation image corresponding to positions of the selected edge-pixels in the edge image; (Tribelhorn, paragraph 0062, “The resulting digitized image is then filtered by a set of four filters 250, 260, 270, 180 as shown in FIG. 5 having the combined effect of an isotropic filter. The response from these filters 250, 260, 270, 180 is then used as an indication of the likelihood that a set of pixels being an edge. The result of this filtering gives a response for a large section of the original image (24x24-pixel square which is a result of 3x3 pixel square with one-eighth of resolution) [one or more segments representing the one or more digital objects]. FIG. 6A shows an edge response of FIG. 3A derived from the filters of FIG. 5. FIG. 6B shows a schematic drawing of FIG. 6A. The white areas 302 represent stronger response, the gray areas 304 is moderate response, and the black areas 306 represent no response. This conveniently produces a local region of high edge confidence that can be further processed to find exact edges [image includes a plurality of first pixels, positions of the first pixels in the segmentation image corresponding to positions of the selected edge-pixels in the edge image;].”)
selecting one or more second pixels for each first pixel in the segmentation image; (Tribelhorn, paragraph 0061, “In one embodiment, the Sobel operation 31 can be implemented by means of hardware. Yet in another embodiment, the Sobel operation 31 can be implemented by means of software. Only eight image pixels around a pixel [selecting one or more second pixels] are needed to compute the corresponding result and only simple integer mathematics is needed to compute the gradient vector approximation.” And paragraph 0062, “In the present embodiment, a naive threshold is used to generate binary image (i.e., to set a pixel to be a 1 or a 0 depending on whether the pixel pass the threshold value.”) [for each first pixel in the segmentation image])
Tribelhorn does not explicitly teach generating a distorted segmentation image using a two-dimensional distortion applied to the segmentation image, wherein the two-dimensional distortion determines a pixel value of each pixel in the distorted segmentation image using the first pixels and the second pixels; and
However, Tang teaches generating a distorted segmentation image using a two-dimensional distortion applied to the segmentation image, wherein the two-dimensional distortion determines a pixel value of each pixel in the distorted segmentation image using the first pixels and the second pixels; and (Tang, pg. 133119, Col. 2, paragraph 2, “Therefore, we combined the statistical shape model and the 3D thin plate spline algorithm (3D-TPS) to simulate the generation of 3D medical images. 3D thin plate spline algorithm can give the deformation of the whole space between one real image and one simulated image via the corresponding position of N matching points in the two images.” And paragraph 1, “In addition, this medical image augmentation strategy is also applicable to 2D medical images.”)
Tribelhorn and Tang are related to the same field of endeavor (i.e. image detection). In view of the teachings of Tang, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Tang to Tribelhorn before the effective filing date of the claimed invention in order to heighten the accuracy of image enhancement. (Tang, pg. 133119, Col. 2, paragraph 1, “We think that if our method combines with other image enhancement methods, the accuracy may be higher.”).
Tribelhorn and Tang do not explicitly teach training the generative neural network using the distorted segmentation image as input image to estimate the digital image.
However, Planche teaches training the generative neural network using the distorted segmentation image as input image to estimate the digital image. (Planche, pg. 17, “Each step can be first trained consecutively (i.e. firstly, training of Gseg [Examiner would like to point out that Gseg is a segmented image], then training of Grgb2n while fixing Gseg), then conjointly (training end-to-end).”)
wherein a machine is configured to be controlled based on the estimated digital image. (Planche, pg. 17, paragraph 1, “The task specific loss is then the distance between the estimations (recovered information) of the recognition method on the original data [controlled based on the estimated digital image], and on the generated ones. In other words, it guides the GAN [a machine] to generate normal maps which would induce the same responses from the recognition methods as the original clean normal maps.”)
Tribelhorn, Tang and Planche are related to the same field of endeavor (i.e. image detection). In view of the teachings of Planche, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Planche to Tribelhorn and Tang before the effective filing date of the claimed invention in order to identify a desired feature in an image. (Planche, pg. 3, “Hence, the recognition system has the task to identify the desired feature of the object (e.g. class or pose) from the synthetic cluttered color image.”)
In regard to claim 6, Tribelhorn, Tang, and Planche teach the method of claim 1.
Planche further teaches adding a displacement to the position of the first pixel to determine a position of the second pixel. (Planche, pg. 16, paragraph 6, “Normal and foreground generative losses, between its outputs and the expected geometrical maps. A normal generative loss computes the distance between two images (here between the original normal map and the generated one), comparing their pixel values [adding a displacement to the position of the first pixel]. A foreground generative loss computes a similar distance, but ignoring the pixels which don't belong to the foreground object (using a binary mask).”)
Tribelhorn and Planche are combinable for the same rationale as set forth above with respect to claim 1.
In regard to claim 7, Tribelhorn, Tang, and Planche teach the method of claim 1.
Planche further teaches estimating the digital image using the generative neural network applied to the distorted segmentation image; (Planche, pg. 16, “The image generator G consists in a segmentation unit Gseg [the distorted segmentation image] and a color-to-normal unit Grgb2n. The segmentation unit Gseg has the task to extract the foregrounds out of the cluttered images 223. The foreground is sometimes also referred to as the "target object". In other words, the task of the segmentation unit Gseg is to recognize and identify the contour of the object to be analyzed and "cut" it out from the background. In practice, a first convolutional neural network (CNN) is favorably used to accomplish this task [estimating the digital image using the generative neural network].”)
applying a first loss function to the estimated digital image and the digital image to determine a generative loss value; (Planche, pg. 16, “Gseg is trained to convert the cluttered images 223 into a binary mask of their foreground, using a generative loss.”)
applying a second loss function to the estimated digital image and the edge image to determine an edge loss value; and (Planche, pg. 16, “Normal and foreground generative losses, between its outputs and the expected geometrical maps [estimated digital image]. A normal generative loss computes the distance between two images (here between the original normal map and the generated one [the edge image]), comparing their pixel values. A foreground generative loss computes a similar distance, but ignoring the pixels which don't belong to the foreground object (using a binary mask) [applying a second loss function].”)
training the generative neural network to reduce the generative loss value and the edge loss value. (Planche, pg. 17, “As an option (not illustrated in FIG. 2), another CNN, namely Gret, can be used to refine the output of the first CNN Gseg• Gref takes for input the synthetic cluttered images and their respective outputs from G~b2n, using the two modalities to refine the geometrical estimation. It is trained using a generative loss comparing its outputs to the expected maps.”)
Tribelhorn and Planche are combinable for the same rationale as set forth above with respect to claim 1.
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Tribelhorn, in view of Tang and Planche, and in further view of Yao et al (US Published Patent Application No. 20200082198, "Yao").
In regard to claim 2, Tribelhorn, Tang, and Planche teach the method of claim 1.
Planche further teaches generating a training image using the trained generative neural network applied to a training segmentation image; and (Planche, pg. 17, paragraph 5, “After both the recognition unit T and the image generator G are trained [generating a training image], the recognition system T can be used in "real life". During the use phase 230, an unseen, real cluttered image 231 of an object is first given to the image generator G. The image generator G extracts a clean normal map 232 from the cluttered image 231 by, first, extracting the foreground and, second, converting the segmented image into a normal map 232 [applied to a training segmentation image].”)
However, Tribelhorn, Tang, and Planche do not explicitly teach training an image classifier using the generated training image to classify the training image.
Yao teaches training an image classifier using the generated training image to classify the training image. (Yao, paragraph 0196, “In this feedforward CNN network, the output of each convolution layers passes to the next layer to a final prediction layer 1425. At the prediction layer 1425, a classification 1430 can be made such as determining a type of image [training an image classifier] based on the CNN processing and CNN data [using the generated training image to classify the training image.].”)
Tribelhorn, Tang, and Planche and Yao are related to the same field of endeavor (i.e. image detection). In view of the teachings of Yao, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Yao to Tribelhorn, Tang, and Planche before the effective filing date of the claimed invention in order to further increase performance in graphic processing. (Yao, paragraph 0003, “To further increase performance, graphics processors typically implement processing techniques such as pipelining that attempt to process, in parallel, as much graphics data as possible throughout the different parts of the graphics pipeline.”)
In regard to claim 3, Tribelhorn, Tang, and Planche teach the method of claim 1.
Planche further teaches generating a training image using the trained generative neural network applied to a training segmentation image; (Planche, pg. 17, paragraph 5, “After both the recognition unit T and the image generator G are trained [generating a training image], the recognition system T can be used in "real life". During the use phase 230, an unseen, real cluttered image 231 of an object is first given to the image generator G. The image generator G extracts a clean normal map 232 from the cluttered image 231 by, first, extracting the foreground and, second, converting the segmented image into a normal map 232.”)
Tribelhorn and Planche are combinable for the same rationale as set forth above with respect to claim 1.
However, Tribelhorn, Tang, and Planche do not explicitly teach generating a classified image using a trained image classifier applied to the generated training image;
determining a performance of the trained image classifier using the generated classified image and the training segmentation image.
Yao teaches generating a classified image using a trained image classifier applied to the generated training image; (Yao, paragraph 0196, “In this feedforward CNN network, the output of each convolution layers passes to the next layer to a final prediction layer 1425. At the prediction layer 1425, a classification 1430 can be made such as determining a type of image based on the CNN processing and CNN data. [generating a classified image using a trained image classifier]”)
determining a performance of the trained image classifier using the generated classified image and the training segmentation image. (Yao, paragraph 0154, “While the illustrated configuration of the GPGPU 700 can be configured to train neural networks, one embodiment provides alternate configuration of the GPGPU 700 that can be configured for deployment within a high performance or low power inferencing platform b[determining a performance of the trained image classifier].”)
Tribelhorn and Yao are combinable for the same rationale as set forth above with respect to claim 2.
In regard to claim 4, Tribelhorn, Tang, and Planche teach the method of claim 1.
However, Tribelhorn, Tang, and Planche do not explicitly teach selecting the edge-pixels from the plurality of edge pixels using a statistical probability distribution.
Yao teaches selecting the edge-pixels from the plurality of edge pixels using a statistical probability distribution. (Yao, paragraph 0229, “In stage two, the room layout dataset LSUN is fed through the semantic segmentation network, producing pixel-wise 37-channel semantic features. As indoor scene understanding datasets, the model trained on SUNRGBD generalizes well on LSUN. Referring to part (b) of FIG. 17, qualitative results on LSUN are shown, all of which are produced by a softmax operation without post processing techniques like conditional random fields. Then treating every pixel as a sample, a fully connected layer can be learned to bridge the gap between 37-channel semantic features and 4-class edge labels. In order to illustrate that semantic features are discriminative for this task, a standard unsupervised analysis can be implemented with t-sne as described in Reference [24]. Referring to part ( c) in FIG. 17, samples of wall-ceiling edges (we) are shown and wall-floor edges (wf) form obvious clusters in the embedding space. [selecting the edge-pixels from the plurality of edge pixels]. Yet, some samples of wall-wall edges (ww) and background (bg) scatter among each other. In this stage, Y is determined by hidden random variables Z taking values from edge labels as described in References [1] and [4] (set 1). So this fc layer describes the posterior distribution P(ZIY). [using a statistical probability distribution.]”)
Tribelhorn and Yao are combinable for the same rationale as set forth above with respect to claim 2.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Triebelhorn, in view of Tang and Planche and in further view of Beck et al (US Published Patent Application No. 20130223748, "Beck").
In regard to claim 5, Tribelhorn, Tang, and Planche teach the method of claim 1.
Tribelhorn, Tang, and Planche do not explicitly teach wherein the two-dimensional distortion applied to the segmentation image includes a thin-plate spline transformation.
However, Beck further teaches wherein the two-dimensional distortion applied to the segmentation image includes a thin-plate spline transformation. (Beck, paragraph 0035, “The transformation functions are most preferably so-called 'thin-plate spline' transformations in each case.”)
Tribelhorn, Tang, Planche and Beck are related to the same field of endeavor (i.e. image detection). In view of the teachings of Beck, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Beck to Tribelhorn, Tang and Planche before the effective filing date of the claimed invention in order to allow for great accuracy of object detection. (Beck, paragraph 0189, “It is clear from the foregoing description that a boundary surface network of a tubular object can be determined very rapidly and with great accuracy of (locally specific) detail by way of embodiments of the invention.”)
Claims 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Tribelhorn, in view of Tang and Planche and in furhter view of Mallet (US Published Patent Application No. 20160180501).
In regard to claim 10, and analogous claims 11 and 12 Tribelhorn, Tang, and Planche teach the method of claim 1.
However, Tribelhorn, Tang and Planche do not explicitly teach wherein a number of pixels in the distorted segmentation image is equal to a number of pixels in the segmentation image.
Mallet teaches wherein a number of pixels in the distorted segmentation image is equal to a number of pixels in the segmentation image. (Mallet, paragraph 0061, “The result of the mapping 540 is to produce output coordinate locations and a distorted output segmentation of the reference image field 530 into reference segments, wherein each output coordinate location, with respect to the (x,y)-coordinate system, nevertheless retains the associated enumeration value corresponding to its source segment in the initial image field. This data is stored as the segmented rendered distortion image (SRDI).”)
Tribelhorn, Tang, Planche and Mallet are related to the same field of endeavor (i.e. image detection). In view of the teachings of Mallet, it would have been obvious for a person with ordinary skill in the art to apply the teachings of Mallet to Tribelhorn, Tang and Planche before the effective filing date of the claimed invention in order to efficiently distort an image. (Mallet, paragraph 0005, “Thereafter, when the corrected image ( or an edited version of the corrected image) needs to be re-distorted, the lookup table can be accessed to identify the entry for the corrected image and values stored in the lookup table are used to quickly and efficiently re-distort the image.”)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SKYLAR K VANWORMER whose telephone number is (703)756-1571. The examiner can normally be reached M-F 6:00am to 3:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.K.V./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights