DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
A rejection based on double patenting of the “same invention” type finds its support in the language of 35 U.S.C. 101 which states that “whoever invents or discovers any new and useful process... may obtain a patent therefor...” (Emphasis added). Thus, the term “same invention,” in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957).
A statutory type (35 U.S.C. 101) double patenting rejection can be overcome by canceling or amending the claims that are directed to the same invention so they are no longer coextensive in scope. The filing of a terminal disclaimer cannot overcome a double patenting rejection based upon 35 U.S.C. 101.
Claims 1-20 is/are rejected under 35 U.S.C. 101 as claiming the same invention as that of claims 1-20 of prior U.S. Patent No. 12165348. This is a statutory double patenting rejection.
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-17 of U.S. Patent No. 11694354. Although the claims at issue are not identical, they are not patentably distinct from each other because claims of instant application are anticipated by claims of U.S. Patent No. 11694354.
Claim 1 of instant application
Claim 1 of U.S. Patent No. 11694354
A non-transitory computer readable medium storing computer executable code that when executed by one or more computer processors causes the one or more computer processors to: receive an image of a structure, the image having pixels with first pixel values depicting the structure and second pixel values outside of the structure depicting a background of a geographic area surrounding the structure, and image metadata including first geolocation data; generate a synthetic shape image of the structure from the image using a machine learning algorithm, the synthetic shape image including pixels having pixel values forming a synthetic shape which is a wireframe outline of the structure, the synthetic shape image having second geolocation data derived from the first geolocation data; and map the wireframe outline onto the image of the structure, based at least in part on the first and second geolocation data.
A non-transitory computer readable medium storing computer executable code that when executed by one or more computer processors causes the one or more computer processors to: receive an image of a structure having an outline, the image having pixels with first pixel values depicting the structure and second pixel values outside of the structure depicting a background of a geographic area surrounding the structure, and image metadata including first geolocation data; generate a synthetic shape image of the structure from the image using a machine learning algorithm, the synthetic shape image including pixels having pixel values forming a synthetic shape of the outline, the synthetic shape image having second geolocation data derived from the first geolocation data, wherein the synthetic shape of the outline is a vector outline of the structure; map the vector outline onto the image of the structure, based at least in part on the first and second geolocation data; and change the second pixel values of the image so as to not depict the background of the geographic area surrounding the structure.
Claim 11 of instant application
Claim 10 of U.S. Patent No. 11694354
A non-transitory computer readable medium storing computer executable code that when executed by one or more computer processors causes the one or more computer processors to: apply a first machine learning algorithm and a second machine learning algorithm to a plurality of truth pairs, each of the truth pairs including a truth image and a truth shape image, the truth image having first pixel values depicting a structure and second pixel values depicting a background of a geographic area surrounding the structure, the structure having an outline, the truth shape image having third pixel values indicative of a truth shape indicating the outline of the structure; generate a synthetic shape image of the structure from the truth image using the first machine learning algorithm, the synthetic shape image including pixels having fourth pixel values forming a synthetic shape of the outline of the structure; pass the synthetic shape image of the structure from the first machine learning algorithm to the second machine learning algorithm; compare the synthetic shape against a truth shape from the truth shape image; and provide feedback from the second machine learning algorithm to the first machine learning algorithm to train the first machine learning algorithm to minimize any differences in the synthetic shape and the truth shape.
A method, comprising: supplying a plurality of truth pairs to a first machine learning algorithm and a second machine learning algorithm, each of the truth pairs including a truth image and a truth shape image, the truth image having first pixel values depicting a structure and second pixel values depicting a background of a geographic area surrounding the structure, the structure having an outline, the truth shape image having third pixel values indicative of a truth shape indicating the outline of the structure, wherein the truth image and the truth shape image have a same pixel resolution; generating a synthetic shape image of the structure from the truth image using the first machine learning algorithm, the synthetic shape image including pixels having fourth pixel values forming a synthetic shape of the outline of the structure; passing the synthetic shape image of the structure from the first machine learning algorithm to the second machine learning algorithm; comparing the synthetic shape against a truth shape from the truth shape image; and providing feedback from the second machine learning algorithm to the first machine learning algorithm to train the first machine learning algorithm to minimize any differences in the synthetic shape and the truth shape.
Claim 14 of instant application
Claim 12 of U.S. Patent No. 11694354
A method, comprising: receiving, with one or more computer processors, an image of a structure, the image having pixels with first pixel values depicting the structure and second pixel values outside of the structure depicting a background of a geographic area surrounding the structure, and image metadata including first geolocation data; generating a synthetic shape image of the structure from the image using a machine learning algorithm, the synthetic shape image including pixels having pixel values forming a synthetic shape which is a wireframe outline of the structure, the synthetic shape image having second geolocation data derived from the first geolocation data; and mapping the wireframe outline onto the image of the structure, based at least in part on the first and second geolocation data.
A method, comprising: receiving, with one or more computer processors, an image of a structure having an outline, the image having pixels with first pixel values depicting the structure and second pixel values outside of the structure depicting a background of a geographic area surrounding the structure, and image metadata including first geolocation data; generating a synthetic shape image of the structure from the image using a machine learning algorithm, the synthetic shape image including pixels having pixel values forming a synthetic shape of the outline, the synthetic shape image having second geolocation data derived from the first geolocation data, wherein the synthetic shape of the outline is a vector outline of the structure; mapping, with the one or more computer processors, the vector outline onto the image of the structure, based at least in part on the first and second geolocation data; and changing the second pixel values of the image so as to not depict the background of the geographic area surrounding the structure.
Claims 2-10, 12, 13 and 15-20 are rejected on the ground of nonstatutory obviousness-type double patenting as being dependent upon a rejected base claim, but would be withdrawn from the rejection if their base claims overcome the rejection by the timely filing of a terminal disclaimer.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-10 and 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Estrada (U.S. PG-PUB NO. 2018/0158210) in view of Yang (U.S. PATENT NO. 10997464).
-Regarding claim 1, Estrada discloses a non-transitory computer readable medium storing computer executable code that when executed by one or more computer processors causes the one or more computer processors (FIG. 1-4) to: receive an image of a structure, the image having pixels with first pixel values depicting the structure and second pixel values outside of the structure depicting a background of a geographic area surrounding the structure, and image metadata including first geolocation data (“orthorectified geospatial image” refers to satellite imagery of the earth that has been digitally corrected to remove terrain distortions introduced into the image by either angle of incidence of a particular point from the center of the satellite imaging sensor or significant topological changes inherent to the region of the earth that the image depicts, [0033]; building, [0042]; background, [0062]; metadata, [0068]); and map the wireframe outline onto the image of the structure, based at least in part on the first and second geolocation data (synthetically manipulated geospatial image 1303 showing synthetic image 1302 scaled, aligned, masked and overlain onto geospatial image 1301 merged into synthetic image overlay area 1330, [0076]; Establishment of the location of any identified objects of interest generally requires that conversion of the coordinate system used internal to the cache of multi-scale unanalyzed geospatial image segments 530 to the coordinates of earth latitude and longitude takes place, [0072]).
Estrada is silent to teaching that generate a synthetic shape image of the structure from the image using a machine learning algorithm, the synthetic shape image including pixels having pixel values forming a synthetic shape which is a wireframe outline of the structure, the synthetic shape image having second geolocation data derived from the first geolocation data. However, the claimed limitation is well known in the art as evidenced by Yang.
In the same field of endeavor, Yang teaches generate a synthetic shape image of the structure from the image using a machine learning algorithm (machine learning, col. 3 lines 42-57), the synthetic shape image including pixels having pixel values forming a synthetic shape which is a wireframe outline of the structure, the synthetic shape image having second geolocation data derived from the first geolocation data (the wireframe rendering discriminator module rasterizes the refined training digital image layout to form a wireframe digital image layout such that the graphic elements are converted into two-dimensional wireframe images, col. 4 lines 23-41; pixel I(Xt,Yt) at the location (Xt,Yt) in the rendered image may be calculated through class-wise maximum operation of the rendered class probability distribution on (Xt,Yt) of each graphic element, col. 11 lines 22-46).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Estrada with the teaching of Yang in order to optimize layout.
-Regarding claim 2, the combination further discloses the wireframe outline includes edges and nodes defining an outline of the structure (Yang, points near the boundary of the dotted box (lie in the solid line), col. 12 lines 1-5).
-Regarding claim 3, the combination further discloses the computer executable code that when executed by the one or more computer processors further causes the one or more computer processors to utilize the wireframe outline on the image of the structure to isolate the first pixel values depicting the structure (Estrada, the image manipulation software module 1020 separates the synthetic image layer from the real image layer; demarcated synthetic image 1180 is overlain onto existing real imagery background, and using a masking function 1185 to set the background of the synthetic image to transparent such that existing imagery is not occluded, [0074]).
-Regarding claim 4, the combination further discloses the computer executable code that when executed by the one or more computer processors further causes the one or more computer processors to change the second pixel values of the image so as to not depict the background of the geographic area outside of the wireframe outline of the structure (Estrada, using a masking function 1185 to set the background of the synthetic image to transparent such that existing imagery is not occluded, [0074]; the filter changes all pixels constituent to the cloud to a single color value 702 to clearly demarcate the portion of the image that is obstructed, [0071]).
-Regarding claims 5, 6 and 17, although the combination does not specifically disclose image has a pixel resolution less than 10 inches per pixel or between 10 inches per pixel and 0.1 inches per pixel, the examiner takes official notice that pixel resolution is merely a design preference. Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the pixel resolution to be less than 10 inches per pixel or between 10 inches per pixel and 0.1 inches per pixel based on design choices.
-Regarding claim 7, the combination further discloses the machine learning algorithm is a first machine learning algorithm, and wherein the first machine learning algorithm is a component of a generator of a generative adversarial network (Yang, generative adversarial network (GAN) system 122 includes a generator module 208 having a self-attention module 210, col. 8 lines 42-49), the generative adversarial network further comprising a discriminator having a second machine learning algorithm, the generator receiving the image of the structure and generating the synthetic shape image (Yang, The first discriminator module is a wireframe rendering discriminator module 214 that is configured to optimize the refined training digital image in a visual domain using wireframes, col. 8 lines 50-57).
-Regarding claim 8, the combination further discloses the generative adversarial network has been trained with truth pairs with each truth pair including a truth image and a truth shape image (Yang, a plurality of ground truth digital image layouts, i.e., a set of digital image layouts that are considered visually pleasing, col. 4 lines 42-63; layout optimization module 410, for instance, may select a set of random samples from real data maintained in a storage device as a ground truth 414 and employ a convolutional neural network 412 to recognize a distribution exhibited by the ground truth 414, col. 11 lines 4-21).
-Regarding claim 9, the combination further discloses the truth image and the truth shape image have a same pixel resolution (Yang, A graphic layout with N graphic elements is to be rasterized (denoted as {(pi,θi), . . . , (pN,θN)}) onto a target image I(Xt,Yt) layout, where (Xt,Yt) is the location in a pre-defined regular grid, col. 11 lines 22-46).
-Regarding claim 10, the combination further discloses the image is a nadir image (Estrada, orthorectified geospatial image, [0033]).
-Regarding claim 14, Estrada discloses a method, comprising: receiving, with one or more computer processors, an image of a structure, the image having pixels with first pixel values depicting the structure and second pixel values outside of the structure depicting a background of a geographic area surrounding the structure, and image metadata including first geolocation data (“orthorectified geospatial image” refers to satellite imagery of the earth that has been digitally corrected to remove terrain distortions introduced into the image by either angle of incidence of a particular point from the center of the satellite imaging sensor or significant topological changes inherent to the region of the earth that the image depicts, [0033]; building, [0042]; background, [0062]; metadata, [0068]); and mapping the wireframe outline onto the image of the structure, based at least in part on the first and second geolocation data (synthetically manipulated geospatial image 1303 showing synthetic image 1302 scaled, aligned, masked and overlain onto geospatial image 1301 merged into synthetic image overlay area 1330, [0076]; Establishment of the location of any identified objects of interest generally requires that conversion of the coordinate system used internal to the cache of multi-scale unanalyzed geospatial image segments 530 to the coordinates of earth latitude and longitude takes place, [0072]).
Estrada is silent to teaching generating a synthetic shape image of the structure from the image using a machine learning algorithm, the synthetic shape image including pixels having pixel values forming a synthetic shape which is a wireframe outline of the structure, the synthetic shape image having second geolocation data derived from the first geolocation data. However, the claimed limitation is well known in the art as evidenced by Yang.
In the same field of endeavor, Yang teaches generating a synthetic shape image of the structure from the image using a machine learning algorithm (machine learning, col. 3 lines 42-57), the synthetic shape image including pixels having pixel values forming a synthetic shape which is a wireframe outline of the structure, the synthetic shape image having second geolocation data derived from the first geolocation data (the wireframe rendering discriminator module rasterizes the refined training digital image layout to form a wireframe digital image layout such that the graphic elements are converted into two-dimensional wireframe images, col. 4 lines 23-41; pixel I(Xt,Yt) at the location (Xt,Yt) in the rendered image may be calculated through class-wise maximum operation of the rendered class probability distribution on (Xt,Yt) of each graphic element, col. 11 lines 22-46).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Estrada with the teaching of Yang in order to optimize layout.
-Regarding claim 15, the combination further discloses the wireframe outline includes edges and nodes defining an outline of the structure (Yang, points near the boundary of the dotted box (lie in the solid line), col. 12 lines 1-5).
-Regarding claim 16, the combination further discloses changing the second pixel values of the image so as to not depict the background of the geographic area outside of the wireframe outline of the structure (Estrada, using a masking function 1185 to set the background of the synthetic image to transparent such that existing imagery is not occluded, [0074]; the filter changes all pixels constituent to the cloud to a single color value 702 to clearly demarcate the portion of the image that is obstructed, [0071]).
-Regarding claim 18, the combination further discloses the machine learning algorithm is a first machine learning algorithm, and wherein the first machine learning algorithm is a component of a generator of a generative adversarial network (Yang, generative adversarial network (GAN) system 122 includes a generator module 208 having a self-attention module 210, col. 8 lines 42-49), the generative adversarial network further comprising a discriminator having a second machine learning algorithm, the generator receiving the image of the structure and generating the synthetic shape image (Yang, The first discriminator module is a wireframe rendering discriminator module 214 that is configured to optimize the refined training digital image in a visual domain using wireframes, col. 8 lines 50-57).
-Regarding claim 19, the combination further discloses the generative adversarial network has been trained with truth pairs with each truth pair including a truth image and a truth shape image (Yang, a plurality of ground truth digital image layouts, i.e., a set of digital image layouts that are considered visually pleasing, col. 4 lines 42-63; layout optimization module 410, for instance, may select a set of random samples from real data maintained in a storage device as a ground truth 414 and employ a convolutional neural network 412 to recognize a distribution exhibited by the ground truth 414, col. 11 lines 4-21).
-Regarding claim 20, the combination further discloses the truth image includes third geolocation data and the truth shape image includes fourth geolocation data, the fourth geolocation data being derived from the third geolocation data (Estrada, the only stipulation being that there is access to a cache of orthorectified, multi-scale, geospatial image segments tagged with information that allows the geographical location of image capture and the scaling factor to be determined 530, [0063]).
Claim(s) 11-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yang (U.S. PATENT NO. 10997464) in view of Estrada (U.S. PG-PUB NO. 2018/0158210).
-Regarding claim 11, Yang discloses a non-transitory computer readable medium storing computer executable code that when executed by one or more computer processors causes the one or more computer processors (FIG. 11) to: apply a first machine learning algorithm and a second machine learning algorithm to a plurality of truth pairs, each of the truth pairs including a truth image and a truth shape image, the truth image having first pixel values and second pixel values, the truth shape image having third pixel values indicative of a truth shape (in a GAN system 122 a discriminator is used to distinguish between synthetic layouts generated by the generator module 208 and real layouts, e.g., ground truths, col. 10 lines 48-61; These plurality of ground truth digital images layouts are also rasterized to form wireframes of graphic elements contained within the layouts, col. 4 lines 42-63); generate a synthetic shape image of the structure from the truth image using the first machine learning algorithm, the synthetic shape image including pixels having fourth pixel values (generator module 208 is configured to generate the refined training digital image layout 212 from the training digital image layout 204 through group relational modeling of graphic elements 206, col. 9 lines 33-40); pass the synthetic shape image of the structure from the first machine learning algorithm to the second machine learning algorithm (refined digital image layout 212 is rasterized by a rasterization module 406 into a wireframe digital image layout 404 of the wireframe rendering discriminator module (block 608), e.g., to form two-dimensional wireframe rendering of graphic elements in the refined training digital image layout 212, col. 11 lines 4-21); compare the synthetic shape against a truth shape from the truth shape image ( wireframe digital image layout is then compared with at least one ground truth digital image layout using a loss function as part of machine learning by the wireframe discriminator module, col. 4 lines 23-41); and provide feedback from the second machine learning algorithm to the first machine learning algorithm to train the first machine learning algorithm to minimize any differences in the synthetic shape and the truth shape (gradients can be propagated backward to both the semantic parameters (e.g., class probabilities) and geometric parameters of the graphic elements for joint optimization, col. 12 lines 35-51).
Yang is silent to teaching that depicting a structure; depicting a background of a geographic area surrounding the structure, the structure having an outline, and indicating the outline of the structure; forming a synthetic shape of the outline of the structure. However, the claimed limitation is well known in the art as evidenced by Estrada.
In the same field of endeavor, Estrada teaches depicting a structure; depicting a background of a geographic area surrounding the structure, the structure having an outline, and indicating the outline of the structure; forming a synthetic shape of the outline of the structure (“orthorectified geospatial image” refers to satellite imagery of the earth that has been digitally corrected to remove terrain distortions introduced into the image by either angle of incidence of a particular point from the center of the satellite imaging sensor or significant topological changes inherent to the region of the earth that the image depicts, [0033]; building, [0042]; background, [0062]; metadata, [0068]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Yang with the teaching of Estrada in order to automate the otherwise manual demarcation of objects in large image corpora.
-Regarding claim 12, the combination further discloses the first machine learning algorithm is a component of a generator of a generative adversarial network (Yang, generative adversarial network (GAN) system 122 includes a generator module 208 having a self-attention module 210, col. 8 lines 42-49), and the second machine learning algorithm is a component of a discriminator (Yang, The first discriminator module is a wireframe rendering discriminator module 214 that is configured to optimize the refined training digital image in a visual domain using wireframes, col. 8 lines 50-57).
-Regarding claim 13, the combination further discloses the truth image and the truth shape image have a same pixel resolution (Yang, A graphic layout with N graphic elements is to be rasterized (denoted as {(pi,θi), . . . , (pN,θN)}) onto a target image I(Xt,Yt) layout, where (Xt,Yt) is the location in a pre-defined regular grid, col. 11 lines 22-46).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING Y HSIEH whose telephone number is (571)270-3011. The examiner can normally be reached Monday-Friday, 9am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING Y HSIEH/ Primary Examiner, Art Unit 2664
Prosecution Insights