DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . The preliminary Amendment filed 15 November 2023 has been entered and considered. Claims 1-10 and 13-18 have been amended. Claims 1-18 are all the claims pending in the application.
Priority
This application is the national stage of PCT/SG2022/050350, filed May 25, 2022. This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of Application No. SG10202107190U, filed in Singapore on 06/30/2021.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 11/15/2023 and 12/27/2023 were considered by the examiner.
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the steps of the method recited in independent Claim 1 (corresponding to boxes in Fig. 1) and Claim 14 (corresponding to boxes in Figs. 8 and 9) must be shown or the features canceled from the claims. In particular, Figs. 1, 8, and 9 depict a flow diagram without “readily identifiable” descriptors of each block, as required by 37 CFR 1.84(n). Rule 84(n) requires “labeled representations” of graphical symbols, such as blocks; and any that are “not universally recognized may be used, subject to approval by the Office, if they are not likely to be confused with existing conventional symbols, and if they are readily identifiable”. In the case of Figs. 1, 8, and 9, the blocks are not readily identifiable per se and therefore require the insertion of text that identifies the function of that block. No new matter should be entered.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 18 is rejected under 35 U.S.C. 101 because it is directed to a computer program per se. Computer programs claimed as computer listings per se, i.e., the descriptions or expressions of the programs, are not physical “things.” They are neither computer components nor statutory processes, as they are not “acts” being performed. Such claimed computer programs do not define any structural and functional interrelationships between the computer program and other claimed elements of a computer which permit the computer program’s functionality to be realized (See MPEP § 2106.01(I)).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Claim 1 recites the limitations “image storage unit” and “determination unit”. These limitations have been interpreted under 112(f) as a means plus function because of the combination of the non-structural, generic placeholder “image storage unit” and “determination unit”, as well as their respective functional languages “that stores” and “that obtains… and determines” and is being interpreted respectively as “a network or an external storage medium such as a USB memory” and “learned model 22 is a trained model… on an external terminal device such as a personal computer” that corresponds to the structure found in the disclosure (Par. [0045]).
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-9, 13, and 18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Pan et al. (NPL: Conditional Generative Adversarial Network-Based Training Sample Set Improvement Model for the Semantic Segmentation of High-Resolution Remote Sensing Images, hereafter referred as Pan).
Regarding Claim 1:
Pan teaches a computer-implemented training method of training a map segmenter comprising a deep neural network (Pan: Abstract; conditional generative adversarial network (CGAN)-based training sample set improvement model (CGAN-TSIM) for the semantic segmentation of high-resolution remote sensing images), comprising: providing a training dataset (TDS1) comprising training image data comprising training pairs (TDP1) of map images of a geographical area (GA1) acquired by one or more image acquisition apparatuses (SAT1) and corresponding segmentation masks, wherein the training image data is stored in a computer memory (CM1) (Pan: A. Overall Process of the Proposed Method and Fig. 4; there are Nimg remote sensing images and corresponding ground-truth images; each pair consisting of an image and its ground truth is cut into training image patches; these training image patches form the original training sample set; the samples in the sample set are used to train a CGAN); generating synthetic map images by creating synthetic map images by applying a generative adversarial network (GAN) onto segmentation masks, wherein the segmentation masks comprises the corresponding segmentation masks and additional segmentation masks (Pan: A. Overall Process of the Proposed Method and Fig. 1; generate ground-truth images, improve the diversity of the training samples; the generator G in the CGAN takes MC as the input conditions and generates a set of corresponding remote sensing images, MG); storing the synthetic map images and the corresponding additional segmentation masks as additional training data pairs (TDP2) in the training dataset (TDS1) in the computer memory (CM1) (Pan: A. Overall Process of the Proposed Method and Fig. 1; improve the diversity of the training samples; MC and MG are combined to form a new training set snew−i; the set snew−i and the set si are combined to form a new set, sei = snew−i ∪ si , where sei consists of the improved training samples corresponding to the i th remote sensing image); and training the map segmenter with the training dataset (TDS1) (Pan: 4) CGAN-TSIM; the improved training sample set obtained by applying CGAN-TSIM to the image patches obtained via the DCS was used).
In regards to Claim 2, Pan further teaches the training method of claim 1, wherein the additional segmentation masks are provided by: a segmentation mask database and wherein the additional segmentation masks are different from the corresponding segmentation masks corresponding to the map images; or a segmentation mask generator configured to generate a representation of a road network and transform the representation into the additional segmentation mask (Pan: C. Methods Considered for Comparison and Fig. 3; the training and ground-truth images need to be cut into fixed-size image patches before they can be input into the neural network; four strategies were adopted to obtain the training image patches).
In regards to Claim 3, Pan further teaches the training method of claim 1, wherein the generative adversarial network (GAN) is trained by training: a generator model (G1) with a segmentation mask of the segmentation masks; and a discriminator model (Dl) configured to discriminate between the image generated by the generator model (G1) and a map image corresponding to the segmentation mask (Pan: B. Conditional Generative Adversarial Network for Remote Sensing Image Patch Generation and Fig. 2; the CGAN consists of two modules: a generator (G) and a discriminator (D); the input to G is a ground-truth image patch).
In regards to Claim 4, Pan further teaches the training method of claim 1, wherein creating synthetic map images comprises augmenting the map images by applying the generative adversarial network (GAN) on the corresponding segmentation masks thereby producing augmented images (Pan: A. Overall Process of the Proposed Method and Fig. 1; generate ground-truth images, improve the diversity of the training samples; the generator G in the CGAN takes MC as the input conditions and generates a set of corresponding remote sensing images, MG); and the training method comprises storing the augmented images with their corresponding segmentation masks in the training dataset (TDS1) in the computer memory (Pan: A. Overall Process of the Proposed Method and Fig. 1; improve the diversity of the training samples; MC and MG are combined to form a new training set snew−i; the set snew−i and the set si are combined to form a new set, sei = snew−i ∪ si , where sei consists of the improved training samples corresponding to the i th remote sensing image).
In regards to Claim 5, Pan further teaches the training method of claim 1, wherein creating synthetic map images further comprises: creating a synthetic image of the synthetic map images by applying the generative adversarial network (GAN) onto an additional segmentation mask comprised by the additional segmentation masks (Pan: A. Overall Process of the Proposed Method and Fig. 1; generate ground-truth images, improve the diversity of the training samples; the generator G in the CGAN takes MC as the input conditions and generates a set of corresponding remote sensing images, MG).
In regards to Claim 6, Pan further teaches the training method of claim 1 any of the previous claim, wherein the generative adversarial network (GAN) comprises a conditional-single natural image generative adversarial network (cSinGAN) or a derivate thereof (Pan: 4) Single Training Image is Used for a Clear Sample Generation Assessment and Accuracy Comparison; the use of a single training image allows the sample generation ability of CGAN-TSIM to be observed more clearly).
In regards to Claim 7, Pan further teaches the training method of claim 1, wherein the generative adversarial network (GAN) comprises a Multi-Categorical-conditional-single natural image generative adversarial network or a derivate thereof (Pan: 2) Object Placement Strategy and Pseudo-Algorithm 3; the object size and shape are fixed, but changing tθ and td will result in different subimages, producing a variety of ground-truth images).
In regards to Claim 8, Pan further teaches the training method of claim 6, wherein the cSinGAN comprises two or more generators (CAT=1, CAT=2), each trained for a category of the categories; and the training method, further comprises, for each generator (CAT=1, CAT=2): inputting a noise tensor into the cSinGAN (Pan: 2) Object Placement Strategy and Pseudo-Algorithm 3; the object size and shape are fixed, but changing tθ and td will result in different subimages, producing a variety of ground-truth images; these two parameters lead to diverse cropping results and the creation of ground-truth images with rich spatial diversity).
In regards to Claim 9, Pan further teaches the training method of claim 7, wherein the Multi-Categorical-cSinGAN comprises a multi-scale generator set, and the training method further comprises selecting a noise section from a region of a noise space , the noise region corresponding to a given category of the categories, wherein the noise section is randomly selected within the region; and inputting the noise section as noise tensor into the multi-scale generator set (Pan: 2) Object Placement Strategy and Pseudo-Algorithm 3; the object size and shape are fixed, but changing tθ and td will result in different subimages, producing a variety of ground-truth images; these two parameters lead to diverse cropping results and the creation of ground-truth images with rich spatial diversity).
In regards to Claim 13, Pan further teaches the training method of claim 1, wherein training the map segmenter with the training dataset (TDS1) comprises at least 2, for example 3, training phases (Pan: 4) CGAN-TSIM; to analyze the impact of different training samples on a semantic segmentation neural network, we adopted the ordinary U-Net model as the semantic segmentation model; based on U-Net and the four patch strategies, we compared five methods in our experiments), wherein: at least one of the training phases is performed with training image data comprising the training pairs (TDP1) and without of additional training data pairs (TDP2); and at least another one of the training phases is performed with the additional training data pairs (TDP2) (Pan: 4) CGAN-TSIM; U-Net + DCS, only the image patches obtained via the DCS were used as the training sample set; U-Net + CGAN-TSIM, the improved training sample set obtained by applying CGAN-TSIM to the image patches obtained via the DCS was used).
Regarding Claim 18:
Pan further teaches a computer program product comprising program instructions, which when executed by one or more processors, cause the one or more processors to perform a method of training a map segmenter comprising a deep neural network (Pan: Abstract; conditional generative adversarial network (CGAN)-based training sample set improvement model (CGAN-TSIM) for the semantic segmentation of high-resolution remote sensing images), the method comprising: providing a training dataset (TDS1) comprising training image data comprising training pairs (TDP1) of map images of a geographical area (GA1) acquired by one or more image acquisition apparatuses (SAT1) and corresponding segmentation masks, wherein the training image data is stored in a computer memory (CM1) (Pan: A. Overall Process of the Proposed Method and Fig. 4; there are Nimg remote sensing images and corresponding ground-truth images; each pair consisting of an image and its ground truth is cut into training image patches; these training image patches form the original training sample set; the samples in the sample set are used to train a CGAN); generating synthetic map images by creating synthetic map images by applying a generative adversarial network (GAN) onto segmentation masks, wherein the segmentation masks comprises the corresponding segmentation masks and additional segmentation masks (Pan: A. Overall Process of the Proposed Method and Fig. 1; generate ground-truth images, improve the diversity of the training samples; the generator G in the CGAN takes MC as the input conditions and generates a set of corresponding remote sensing images, MG); storing the synthetic map images and the corresponding additional segmentation masks as additional training data pairs (TDP2) in the training dataset (TDS1) in the computer memory (CM1) (Pan: A. Overall Process of the Proposed Method and Fig. 1; improve the diversity of the training samples; MC and MG are combined to form a new training set snew−i; the set snew−i and the set si are combined to form a new set, sei = snew−i ∪ si , where sei consists of the improved training samples corresponding to the i th remote sensing image); and training the map segmenter with the training dataset (TDS1) (Pan: 4) CGAN-TSIM; the improved training sample set obtained by applying CGAN-TSIM to the image patches obtained via the DCS was used).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Pan et al. (NPL: Conditional Generative Adversarial Network-Based Training Sample Set Improvement Model for the Semantic Segmentation of High-Resolution Remote Sensing Images, hereafter referred as Pan) in view of Li et al. (C.N. Patent Pub No. 109086668 A, hereafter referred as Li).
In regards to Claim 14, Pan fails to further teach a computer-implemented segmenting method for extracting a road network for use in vehicle routing, the segmenting method comprising: providing a trained segmenter trained by using the training dataset of the training method of claim 1; providing processing image data comprising map images acquired by one or more image acquisition devices; segmenting, by the trained segmenter, each of the map images thereby determining attributes to different portions of the image; storing the segmented images and/or the attributes as a road network in a database memory for access by vehicle routing services.
Li, like Pan, is directed to computer-implemented training method of training a map segmenter. Li combined with Pan does teach a computer-implemented segmenting method for extracting a road network for use in vehicle routing (Li: Par. [0001]; method for extracting road information of high resolution unmanned aerial vehicle remote sensing image based on generating confrontation network, and fusing multi-scale image processing), the segmenting method comprising: providing a trained segmenter trained by using the training dataset of the training method of claim 1 (Pan: Abstract; conditional generative adversarial network (CGAN)-based training sample set improvement model (CGAN-TSIM) for the semantic segmentation of high-resolution remote sensing images); providing processing image data comprising map images acquired by one or more image acquisition devices; segmenting, by the trained segmenter, each of the map images thereby determining attributes to different portions of the image (Li: Par. [0050]; independently taking out the generated network generated in the balance state of the generated network to be applied; cutting the remote sensing image shot by the actual unmanned aerial vehicle into a series of remote sensing images of n*n size, and taking it as input; taking the output characteristic image of the generated network as the divided result to extract the road area image); storing the segmented images and/or the attributes as a road network in a database memory for access by vehicle routing services (Li: Par. [0051]; inputting the test data into the generated network; comparing the obtained road area image with the label image; the extracting effect is good).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Pan to utilize the segmenting method for vehicle routing, as taught by Li, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. As taught by Li, the proposed modification would improve the extraction precision of the road area in the unmanned aerial vehicle remote sensing image (Li: Par. [0008]).
In regards to Claim 15, Pan as modified by Li further teaches the method of claim 14, wherein segmenting further comprises, by the trained segmenter classifying pixels of a, or each, image of the map images into road and no-road (Li: Par. [0054] and Fig. 5; the first column image is input of the original unmanned remote sensing image, the second column image is the label image of the corresponding artificial mark, the third column data is fusion multi-scale characteristic of the generated network output of the road area image, the fourth row data is not fused multi-scale characteristic of generating network output of the road area image; by contrast, it can be found that the image background is relatively simple, and the road target area outline is obvious).
Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Pan et al. (NPL: Conditional Generative Adversarial Network-Based Training Sample Set Improvement Model for the Semantic Segmentation of High-Resolution Remote Sensing Images, hereafter referred as Pan) in view of Li et al. (C.N. Patent Pub No. 109086668 A, hereafter referred as Li) and Sun (U.S. Patent App. Pub No. 2021/0042622 A1, hereafter referred as Sun).
In regards to Claim 16, Pan as modified by Li fails to further teach the method of claim 14, further comprising, by a computing system: receiving, by a communication interface, a route request from a user's mobile device; applying a route solver on the route request, on a road map produced from the road network, and a fleet of vehicles, thereby providing a viable route for a vehicle of the fleet; sending the route data of the viable route to the vehicle; receiving an acknowledgement of service from the vehicle; sending the route data, by the communication interface, to the user's mobile device; sending the acknowledgement of service, by the communication interface, to the user's mobile device.
Sun, like Li, is directed to vehicle remote sensing automatic processing technology. Sun does teach receiving, by a communication interface, a route request from a user's mobile device (Sun: Par. [0071]; service requester (e.g., a passenger, a driver, and/or a user) may send and/or transmit the service request to the processing engine 112 through the user terminal 130); applying a route solver on the route request, on a road map produced from the road network, and a fleet of vehicles, thereby providing a viable route for a vehicle of the fleet; sending the route data of the viable route to the vehicle (Sun: Par. [0074]; may determine at least one recommended route for the user terminal based on the service request and the prediction model); receiving an acknowledgement of service from the vehicle; sending the route data, by the communication interface, to the user's mobile device; sending the acknowledgement of service, by the communication interface, to the user's mobile device (Sun: Par. [0076]; the trigger code may be configured to render the operation system of the user terminal 130 to generate a presentation of the conclusion or the result (e.g., the recommended route) on an interface of the user terminal 130).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Pan and Li to utilize the vehicle routing suggestions, as taught by Sun, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. As taught by Li, the proposed modification would guide users to travel along optimal routes with less congestions may improve user experiences and mitigate traffic congestions (Sun: Par. [0003]).
In regards to Claim 17, Pan as modified by Li and Sun further teaches the method of claim 14, further comprising, by a computing system: receiving, by a communication interface, a route request from a vehicle (Sun: Par. [0071]; service requester (e.g., a passenger, a driver, and/or a user) may send and/or transmit the service request to the processing engine 112 through the user terminal 130); applying a route solver on the route request and a road map produced from the road network, thereby providing a viable route for the vehicle (Sun: Par. [0074]; may determine at least one recommended route for the user terminal based on the service request and the prediction model); sending route data of the viable route to the vehicle (Sun: Par. [0076]; the trigger code may be configured to render the operation system of the user terminal 130 to generate a presentation of the conclusion or the result (e.g., the recommended route) on an interface of the user terminal 130).
Allowable Subject Matter
Claims 10-12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 10 recites, wherein the training dataset comprises one or more batches, each batch thereof comprising a segmentation mask and a plurality of synthetic map images generated from the segmentation mask; and wherein the training method further comprises calculating a batch quality score (BQS) for each batch. The cited art of record does not teach or suggest such a combination of features.
Claims 11-12 are allowable subject matter by virtue of their dependency on Claim 10.
Because the cited art of record, alone or in combination, does not teach or suggest each and every feature of dependent Claims 10-12, these claims would be allowable.
Pertinent Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tsutsui et al. (U.S. Patent App. Pub No. 2019/0042888 A1) teaches generating a first network which outputs a saliency map with respect to an input image.
Jaipuria et al. (U.S. Patent App. Pub No. 2021/0142116 A1) teaches generate, via a deep neural network, a first synthetic image based on a simulated image, generate a segmentation mask based on the synthetic image, compare the segmentation mask with a ground truth mask of the synthetic image, update the deep neural network based on the comparison, and generate, via the updated deep neural network, a second synthetic image based on the simulated image.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RENAE BITOR whose telephone number is (703)756-5563. The examiner can normally be reached Monday to Friday: 8:00 - 5:30 but off the 1st Friday of the biweek.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, GREG MORSE can be reached on (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RENAE A BITOR/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights