DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 02/05/2024, 05/28/2024, 08/15/2024, 12/31/2024, 09/10/2025, 12/22/2025, and 01/15/2026 is/are being considered by the Examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-9 and 16-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 7 and 16 recite the limitation "the first vehicles". There is insufficient antecedent basis for this limitation in the claims as the previous recitation in the independent claims is a singular first vehicle.
Claims 8 and 17 recite the limitation "the location". There is insufficient antecedent basis for this limitation in the claims.
Regarding claims 9 and 18, in the third line recite “based on an expected view of at least one with respect to the first vehicle”. There appears to be a missing noun after “at least one” since as currently written there is nothing that would have “an expected view” nor a relationship to the first vehicle. Furthermore, the final line recites “the first camera”. There is insufficient antecedent basis for this limitation in the claims as the previous recitation in the independent claims is a set of cameras.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-5 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhou et al, “Transferring Visual Knowledge for a Robust Road Environment Perception in Intelligent Vehicles” (published in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), October 2017).
Regarding claim 1, Zhou teaches a method comprising:
identifying, by a processor (see Zhou section IV, “The training is implemented using two NVIDIA GTX 1080 GPUs”), a set of images captured by a set of cameras associated with a first vehicle (see section III, B, “In addition to those two datasets, we have also recorded some local traffic scenes around USyd campus. The data collection vehicle is equipped with Point Grey Blackfly cameras and 12.5mm industrial manual lenses”);
for at least one image that depicts an object within the set of images, generating, by the processor, at least one augmented image by modifying at least one visual element of the at least one image (see section III, B, “Since data annotation is prohibitively expensive, some commonly used data augmentation algorithms are used to expand our local dataset and also minimize the occurrence of overfitting [17], [3], [15]. In this paper, we employ four types of image transformations: center cropping, left-right flipping, additive noise and Gaussian blur (shown in Figure 4) to augment the labeled data”); and
training, by the processor, a predictive computer model using a portion of the set of images and the at least one augment image (see section III, C, “The original encoder and decoder of ENet were trained separately. We combine the encoder and decoder so that parameters for the entire network can be fine-tuned together on our local dataset”), wherein the trained predictive computer model is configured to predict a presence of the object in input images (see section IV, C, “In our local traffic environment however, there are more roundabout structures which are considered as ‘undrivable road’. Also, pedestrians (students) usually appear in groups on/around campus with their backpacks or handbags. The classification of these two categories are enhanced after incorporating USyd annotated data into our model. In addition, we had a few frames from a nearby car park building. The results show that even with a small amount of images (less than 10 car park images annotated), the segmentation accuracy can be greatly improved”) for use in autonomous or semi-autonomous control of a second vehicle (see Figure 1, where the “data collection” vehicle is different than the final two vehicles and caption which indicates this is for autonomous vehicles).
Regarding claim 2, Zhou teaches all the limitations of claim 1, and further teaches executing, by the processor, an image manipulation function that maintains a camera property of the at least one image to generate the at least one augmented image (see Zhou Figure 4 which shows “blurring” and “additive noise” which maintains camera properties).
Regarding claim 3, Zhou teaches all the limitations of claim 2, and further teaches wherein the camera property of the at least one image corresponds to at least one of angle, scale, or pose associated with the at least one image (see Zhou Figure 4, wherein “blurring” and “additive noise” would not change any of these features).
Regarding claim 4, Zhou teaches all the limitations of claim 1, and further teaches executing, by the processor, an image manipulation function that adjusts a portion of the at least one image based on a region of interest (see Zhou Figure 4, “cropping” and section III, B, “Instead of random cropping, we crop the center region of images so that objects in the path of the vehicle can be emphasized”).
Regarding claim 5, Zhou teaches all the limitations of claim 4, and further teaches wherein the image manipulation function modifies a camera property corresponding to at least one of cropping, padding, horizontal or vertical flipping, or affine transformations (see Zhou Figure 4, “cropping” and “flipping”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6-8 are is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al, “Transferring Visual Knowledge for a Robust Road Environment Perception in Intelligent Vehicles” (published in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), October 2017) in view of Zhong et al, “Random Erasing Data Augmentation” (published at https://arxiv.org/pdf/1708.04896, November 2017, provided by Applicant on 12/31/2024).
Regarding claim 6, Zhou teaches all the limitations of claim 4, but does not expressively teach wherein the image manipulation function performs at least one of a cutout, hue, saturation, value jitter, salt and pepper, domain transfer, or any combination thereof.
However, Zhong in a similar invention in the same field of endeavor teaches a method of augmenting images containing an object for training a predictive computer model to detect a presence of the object in input images (see Zhong section 4.3, “Object detection aims at detecting instances of semantic objects of a certain class in images. Since the location of each object in the training image is known…”) and executing an image manipulation function that adjusts a portion of the at least one image based on a region of interest (see section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object”) as taught in Zhou wherein
the image manipulation function performs at least one of a cutout (see Zhong section 4.3, “2) Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object” and Figure 2 which shows this is effectively a cutout), hue, saturation, value jitter, salt and pepper, domain transfer, or any combination thereof.
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of manipulating an image based on cutout as taught in Zhong with the method taught in Zhou, the motivation being to utilize the complementary nature of training predictive models with augmented images containing both cropping and cutout (see Zhong section 4.4).
Regarding claim 7, Zhou teaches all the limitations of claim 1, but does not expressively teach wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, wherein a location of the cutout is selected based on an expected view of at least one camera with respect to the first vehicles.
However, Zhong in a similar invention in the same field of endeavor teaches a method of augmenting images containing an object for training a predictive computer model to detect a presence of the object in input images (see Zhong section 4.3, “Object detection aims at detecting instances of semantic objects of a certain class in images. Since the location of each object in the training image is known…”) and executing an image manipulation function that adjusts a portion of the at least one image (see section 4.3, “Since the location of each object in the training image is known, we implement Random Erasing with three schemes”) as taught in Zhou wherein
wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, wherein a location of the cutout is selected (see section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of using a cutout on a selected portion of an image as taught in Zhong with the method taught in Zhou, the motivation being to avoid overfitting while training the predictive model (see Zhang Abstract).
Zhou in view of Zhong further teaches wherein
a location of the cutout is selected based on an expected view of at least one camera with respect to the first vehicles (see Zhong section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object” as combined with Zhou section III, C, “Instead of random cropping, we crop the center region of images so that objects in the path of the vehicle can be emphasized” implying the field of view for the camera shows the path of the vehicle).
Regarding claim 8, Zhou teaches all the limitations of claim 1, but does not expressively teach wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, and wherein the location includes a center of the image that depicts a direction of travel.
However, Zhong in a similar invention in the same field of endeavor teaches a method of augmenting images containing an object for training a predictive computer model to detect a presence of the object in input images (see Zhong section 4.3, “Object detection aims at detecting instances of semantic objects of a certain class in images. Since the location of each object in the training image is known…”) and executing an image manipulation function that adjusts a portion of the at least one image (see section 4.3, “Since the location of each object in the training image is known, we implement Random Erasing with three schemes”) as taught in Zhou wherein
wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, wherein a location of the cutout is selected (see section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of using a cutout on a selected portion of an image as taught in Zhong with the method taught in Zhou, the motivation being to avoid overfitting while training the predictive model (see Zhang Abstract).
Zhou in view of Zhong further teaches wherein
the location includes a center of the image that depicts a direction of travel (see Zhong section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object” as applied to Zhou section III, C, “Instead of random cropping, we crop the center region of images so that objects in the path of the vehicle can be emphasized” i.e. a cutout is applied toward the center of the image which is in the travel direction of the vehicle).
Claim(s) 10-14, 19, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al, “Transferring Visual Knowledge for a Robust Road Environment Perception in Intelligent Vehicles” (published in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), October 2017) in view of Martinez-Canales et al, U.S. Publication No. 2020/0202216.
Regarding claim 10, Zhou teaches a system comprising one or more processors (see Zhou section IV, “The training is implemented using two NVIDIA GTX 1080 GPUs”) to perform operations comprising:
identify a set of images captured by a set of cameras associated with a first vehicle (see section III, B, “In addition to those two datasets, we have also recorded some local traffic scenes around USyd campus. The data collection vehicle is equipped with Point Grey Blackfly cameras and 12.5mm industrial manual lenses”);
for at least one image that depicts an object within the set of images, generate at least one augmented image by modifying at least one visual element of the at least one image (see section III, B, “Since data annotation is prohibitively expensive, some commonly used data augmentation algorithms are used to expand our local dataset and also minimize the occurrence of overfitting [17], [3], [15]. In this paper, we employ four types of image transformations: center cropping, left-right flipping, additive noise and Gaussian blur (shown in Figure 4) to augment the labeled data”);
train a predictive computer model using a portion of the set of images and the at least one augment image (see section III, C, “The original encoder and decoder of ENet were trained separately. We combine the encoder and decoder so that parameters for the entire network can be fine-tuned together on our local dataset”), wherein the trained predictive computer model is configured to predict a presence of the object in input images (see section IV, C, “In our local traffic environment however, there are more roundabout structures which are considered as ‘undrivable road’. Also, pedestrians (students) usually appear in groups on/around campus with their backpacks or handbags. The classification of these two categories are enhanced after incorporating USyd annotated data into our model. In addition, we had a few frames from a nearby car park building. The results show that even with a small amount of images (less than 10 car park images annotated), the segmentation accuracy can be greatly improved”) for use in autonomous or semi-autonomous control of a second vehicle (see Figure 1, where the “data collection” vehicle is different than the final two vehicles and caption which indicates this is for autonomous vehicles).
Zhou does not expressively teach
non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform operations.
However, Martinez-Canales in a similar invention in the same field of endeavor
teaches a system (see Martinez-Canales Figure 1) configured for:
receiving a set of images captured by a set of cameras while affixed to one or
more image collection systems; for each image in the set of images, receiving a
training output for the image (see paragraph [0037]); and training a set of
parameters of a predictive computer model to predict the training output based on an image training set including the images (see paragraph [0038]) as taught
in Zhou wherein the system comprises
non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform operations (see Figure 1 showing the cloud 114, paragraph [0040] which indicates that the steps can be run on a cloud computing distributed model or a physical computing platform executing program instructions, and paragraph [0043] referring to the instructions residing in a memory).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of media storing instructions to be executed by a processor as taught in Martinez-Canales with the system taught in Zhou, the motivation being to automate the method performed by the processor and apply it to different systems.
Independent claim 19 recites similar limitations as claim 10, and is rejected under similar rationale.
Regarding claim 11, Zhou in view of Martinez-Canales teaches all the limitations of claim 10, and further teaches herein the instructions further cause the one or more processors to execute an image manipulation function that maintains a camera property of the at least one image to generate the at least one augmented image (see Zhou Figure 4 which shows “blurring” and “additive noise” which maintains camera properties).
Claim 20 recites similar limitations as claim 11, and is rejected under similar rationale.
Regarding claim 12, Zhou in view of Martinez-Canales teaches all the limitations of claim 11, and further teaches wherein the camera property of the at least one image corresponds to at least one of angle, scale, or pose associated with the at least one image (see Zhou Figure 4, wherein “blurring” and “additive noise” would not change any of these features).
Regarding claim 13, Zhou in view of Martinez-Canales teaches all the limitations of claim 10, and wherein the instructions further cause the one or more processors to execute an image manipulation function that adjusts a portion of the at least one image based on a region of interest (see Zhou Figure 4, “cropping” and section III, B, “Instead of random cropping, we crop the center region of images so that objects in the path of the vehicle can be emphasized”).
Regarding claim 14, Zhou in view of Martinez-Canales teaches all the limitations of claim 13, and further teaches wherein the image manipulation function modifies a camera property corresponding to at least one of cropping, padding, horizontal or vertical flipping, or affine transformations (see Zhou Figure 4, “cropping” and “flipping”).
Claim(s) 15-17 are is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al, “Transferring Visual Knowledge for a Robust Road Environment Perception in Intelligent Vehicles” (published in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), October 2017) in view of Martinez-Canales et al, U.S. Publication No. 2020/0202216 and Zhong et al, “Random Erasing Data Augmentation” (published at https://arxiv.org/pdf/1708.04896, November 2017, provided by Applicant on 12/31/2024).
Regarding claim 15, Zhou in view of Martinez-Canales teaches all the limitations of claim 13, but does not expressively teach wherein the image manipulation function performs at least one of a cutout, hue, saturation, value jitter, salt and pepper, domain transfer, or any combination thereof.
However, Zhong in a similar invention in the same field of endeavor teaches a method of augmenting images containing an object for training a predictive computer model to detect a presence of the object in input images (see Zhong section 4.3, “Object detection aims at detecting instances of semantic objects of a certain class in images. Since the location of each object in the training image is known…”) and executing an image manipulation function that adjusts a portion of the at least one image based on a region of interest (see section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object”) as taught in Zhou in view of Martinez-Canales wherein
the image manipulation function performs at least one of a cutout (see Zhong section 4.3, “2) Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object” and Figure 2 which shows this is effectively a cutout), hue, saturation, value jitter, salt and pepper, domain transfer, or any combination thereof.
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of manipulating an image based on cutout as taught in Zhong with the system taught in Zhou in view of Martinez-Canales, the motivation being to utilize the complementary nature of training predictive models with augmented images containing both cropping and cutout (see Zhong section 4.4).
Regarding claim 16, Zhou in view of Martinez-Canales teaches all the limitations of claim 10, but does not expressively teach wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, wherein a location of the cutout is selected based on an expected view of at least one camera with respect to the first vehicles.
However, Zhong in a similar invention in the same field of endeavor teaches a method of augmenting images containing an object for training a predictive computer model to detect a presence of the object in input images (see Zhong section 4.3, “Object detection aims at detecting instances of semantic objects of a certain class in images. Since the location of each object in the training image is known…”) and executing an image manipulation function that adjusts a portion of the at least one image (see section 4.3, “Since the location of each object in the training image is known, we implement Random Erasing with three schemes”) as taught in Zhou in view of Martinez-Canales wherein
wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, wherein a location of the cutout is selected (see section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of using a cutout on a selected portion of an image as taught in Zhong with the system taught in Zhou in view of Martinez-Canales, the motivation being to avoid overfitting while training the predictive model (see Zhang Abstract).
Zhou in view of Martinez-Canales and Zhong further teaches wherein
a location of the cutout is selected based on an expected view of at least one camera with respect to the first vehicles (see Zhong section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object” as combined with Zhou section III, C, “Instead of random cropping, we crop the center region of images so that objects in the path of the vehicle can be emphasized” implying the field of view for the camera shows the path of the vehicle).
Regarding claim 17, Zhou in view of Martinez-Canales teaches all the limitations of claim 10, but does not expressively teach wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, and wherein the location includes a center of the image that depicts a direction of travel.
However, Zhong in a similar invention in the same field of endeavor teaches a method of augmenting images containing an object for training a predictive computer model to detect a presence of the object in input images (see Zhong section 4.3, “Object detection aims at detecting instances of semantic objects of a certain class in images. Since the location of each object in the training image is known…”) and executing an image manipulation function that adjusts a portion of the at least one image (see section 4.3, “Since the location of each object in the training image is known, we implement Random Erasing with three schemes”) as taught in Zhou in view of Martinez-Canales wherein
wherein the at least one augmented image is generated via performing a cutout applied to the at least one image, wherein a location of the cutout is selected (see section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of using a cutout on a selected portion of an image as taught in Zhong with the system taught in Zhou in view of Martinez-Canales, the motivation being to avoid overfitting while training the predictive model (see Zhang Abstract).
Zhou in view of Martinez-Canales and Zhong further teaches wherein
the location includes a center of the image that depicts a direction of travel (see Zhong section 4.3, “Object-aware Random Erasing (ORE): selecting erasing regions in the bounding box of each object” as applied to Zhou section III, C, “Instead of random cropping, we crop the center region of images so that objects in the path of the vehicle can be emphasized” i.e. a cutout is applied toward the center of the image which is in the travel direction of the vehicle).
Allowable Subject Matter
Claims 9 and 18 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CASEY L KRETZER whose telephone number is (571)272-5639. The examiner can normally be reached M-F 10:00-7:00 PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Payne can be reached at (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CASEY L KRETZER/Primary Examiner, Art Unit 2635
Prosecution Insights