Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/06/2024 was filed and is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “222” has been used to designate both Driving Scenario and Modified Image Data in Fig. 3A; character “224” has been used to designate both Driving Scenario and Modified Image Data in Fig. 3B; and character “226” has been used to designate both Driving Scenario and Modified Image Data in Fig. 3C. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities: Similar to the drawings objection above, character 222, 224, and 226 each are referencing Driving Scenario and Modified Image Data at the same time in Para [0082]-[0084] and [0086]-[0088] .
Appropriate correction is required.
Claim Status
Claims 1-20 are pending in the present application.
Claim 9-14 and 17 are rejected under 35 USC 112(b).
Claims 7-11 are rejected under 35 USC 103 as being unpatentable over Smolyanskiy et al. (US 2021/0150230 A1) in view of BAE et al. (US 2019/0276044 A1).
Claims 15-17 are rejected under 35 USC 103 as being unpatentable over Smolyanskiy et al. (US 2021/0150230 A1) in view of BAE et al. (US 2019/0276044 A1) and YAO (CN 114170826 A).
No prior art rejection is currently applied to claims 1-6, 12-14, and 18-20.
Claims 1-6 and 18-20 are allowed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9-14 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitations “modifying the spatial input scale for the image data comprises modifying image resolution for the image data based on the driving scenario; and modifying the temporal input scale for the image data comprises modifying frame rate for the image data based on the driving scenario”. There is insufficient antecedent basis for this limitation in the claim. Claim 9’s parent claim, claim 8, recites the following limitation: “modifying the input scale for the image data comprises at least one of: modifying spatial input scale…;or modifying temporal input scale…” Since the conditional statement in claim 8 is “or” between “modifying spatial input” and “modifying temporal input scale” with “at least one of” in claim 8’s preamble, there is in insufficient antecedent basis to support both “modifying the spatial input scale” and “modifying the temporal input scale” at the same time in claim 9.
Claim 12 recites the limitations “the first image resolution comprises a greater number of pixels per unit area than the second image resolution; and the first frame rate comprises a greater number of frames per unit time than the second frame rate”. There is insufficient antecedent basis for this limitation in the claim. Claim 12’s parent claim, claim 11, recites the following limitation: “modifying the input scale for the image data comprises at least one of: modifying the image data to a first image resolution…, or modifying the image data to a first frame rate …; and modifying the input scale for the second image data comprises at least one of: modifying the second image data to a second image resolution …, or modifying the second image data to a second frame rate...”. Similar to claim 8 above, claim 11’s preamble for each limitation contains “at least one of” with “or” between limitations. There is lack of antecedent basis for supporting both “modifying to a resolution” and “modifying to a frame rate” for both images at the same time in claim 12
Claim 17 recites the limitation “modifying the spatial input size for the image data comprises modifying a spatial region of interest size for the image data based on the driving scenario; and modifying the temporal input size for the image data comprises modifying a time duration for the image data based on the driving scenario.” There is insufficient antecedent basis for this limitation in the claim. Claim 17’s parent claim, claim 16, recites the following limitation: “modifying the input size for the image data comprises at least one of: modifying a spatial input size for the image data based on the driving scenario; or modifying a temporal input size for the image data based on the driving scenario.” Similar to claim 8 and 11 above, claim 116’s preamble contains “at least one of” with “or” between limitations. There is lack of antecedent basis for supporting both “modifying a spatial input size” and “modifying a temporal input size” at the same time in claim 17.
Claims 10-11 and 13-14 are also rejected for dependent upon either claim 9 or claim 12.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 7-11 are rejected under 35 U.S.C. 103 as being unpatentable over Smolyanskiy et al. (US 2021/0150230 A1, hereinafter Smolyanskiy) in view of BAE et al. (US 2019/0276044 A1, hereinafter Bae).
Regarding claim 7, Smolyanskiy discloses
A method comprising: based on the driving scenario, modifying input scale for image data obtained by the vehicle during the driving scenario (Fig. 5, Para [0047]: “FIG. 5 is a data flow diagram illustrating an example process for pre-processing 404 the sensor data 402 for a machine learning model(s) 408 in an object detection system, in accordance with some embodiments of the present disclosure. The sensor data 402 may be accumulated 510 (which may include transforming to a single coordinate system), ego-motion-compensated 520, and/or encoded 530 into a suitable representation such as a projection image (e.g., a LiDAR range image) and/or a tensor, for example, with multiple channels storing different reflection characteristics”, Para [0048]: “Sensor detections may be accumulated over any desired window of time (e.g., 0.5 seconds (s), 1 s, 2 s, etc.). The size of the window may be selected based on the sensor and/or application (e.g., smaller windows may be selected for noisy applications such as highway scenarios).”, Para [0049]: “In some embodiments, ego-motion-compensation 520 may be applied to the sensor data 402. For example, accumulated detections may be ego-motion-compensated to the latest known vehicle position. More specifically, locations of older detections may be propagated to a latest known position of the moving vehicle, using the known motion of the vehicle to estimate where the older detections will be located (e.g., relative to the present location of the vehicle) at a desired point in time (e.g., the current point in time). The result may be a set of accumulated, ego-motion compensated sensor data 402 ( e.g., a LiDAR point cloud) for a particular time slice.”, Para [0050]: “In some embodiments, the (accumulated, ego-motion compensated) sensor data 402 may be encoded 530 into a suitable representation such as a projection image, which may include multiple channels storing different features such as reflection characteristics.”); and
using a neural network to process the modified image data (Para [0053]: “At a high level, the machine learning model(s) 408 may detect objects such as instances of obstacles, static parts of the environment, and/or other objects represented in the input data 406 ( e.g., a LiDAR range image, camera image, and/or other sensor data stacked into corresponding channels of an input tensor).”).
However, Smolyanskiy does not explicitly disclose
determining a driving scenario for a vehicle.
Bae teaches
determining a driving scenario for a vehicle (Para [0408]: “For example, the processor 270 may determine the traveling state according to the driving environment, as the traveling state in the city road, the traveling state in the highway, parking situation, the curve traveling state, the slope traveling state, the traveling state in the backside road, the traveling state in the off-road, the traveling state in the snow road, the traveling state in the night, the traveling state in the traffic jam, and the like.”, Para [0415]: “As illustrated in FIG. 12B, when it is determined that the vehicle is traveling in a city road, the processor 270 may select AEB, LCA, HBA, LBA, BSD, and automatic parking as the traveling function”, Para [0416]: “When it is determined that the vehicle is traveling in a highway, the processor 270 may select AEB, ACC, LKA, TFA, HBA, LBA, BSD, and automatic parking as the traveling function.”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Smolyanskiy with determining driving scenario and other aspects of Baeas both Smolyanskiy and Bae teaches LiDAR system and analysis of LiDAR images. In addition, modifying Smolyanskiy with determining driving scenario and other aspects of Bae by one of ordinary skill in the art may effectively increase the safety of driving suitable for different driver.
Regarding claim 8, dependent upon claim 7, Smolyanskiy in view of Bae teaches everything regarding claim 7.
Smolyanskiy further teaches
modifying the input scale for the image data comprises at least one of: modifying spatial input scale for the image data based on the driving scenario; or modifying temporal input scale for the image data based on the driving scenario (Para [0048]: “Sensor detections may be accumulated over any desired window of time (e.g., 0.5 seconds (s), 1 s, 2 s, etc.). The size of the window may be selected based on the sensor and/or application (e.g., smaller windows may be selected for noisy applications such as highway scenarios).”).
Regarding claim 9, dependent upon claim 8, Smolyanskiy in view of Bae teaches everything regarding claim 8.
Smolyanskiy further discloses
modifying the spatial input scale for the image data comprises modifying image resolution for the image data based on the driving scenario; and modifying the temporal input scale for the image data comprises modifying frame rate for the image data based on the driving scenario (Para [0048]: “Sensor detections may be accumulated over any desired window of time (e.g., 0.5 seconds (s), 1 s, 2 s, etc.). The size of the window may be selected based on the sensor and/or application (e.g., smaller windows may be selected for noisy applications such as highway scenarios).”; Due to the above 112b rejection, the “and” between the limitations of claim 9 is interpreted as “or”.).
Regarding claim 10, dependent upon claim 8, Smolyanskiy in view of Bae teaches everything regarding claim 8.
Smolyanskiy further discloses
based on the second driving scenario, modifying input scale for second image data obtained by the vehicle during the second driving scenario (Para [0048]: “Sensor detections may be accumulated over any desired window of time (e.g., 0.5 seconds (s), 1 s, 2 s, etc.). The size of the window may be selected based on the sensor and/or application (e.g., smaller windows may be selected for noisy applications such as highway scenarios).”); and
using the neural network to process the modified second image data (Para [0053]: “At a high level, the machine learning model(s) 408 may detect objects such as instances of obstacles, static parts of the environment, and/or other objects represented in the input data 406 ( e.g., a LiDAR range image, camera image, and/or other sensor data stacked into corresponding channels of an input tensor).”, Para [0229]: “The infotaimnent SoC 1630 may include a combination of hardware and software that may be used to provide audio ( e.g., music, a personal digital assistant, navigational instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and/or information services (e.g., navigation systems, rear-parking assistance, a radio data system, vehicle related information such as fuel level, total distance covered, brake fuel level oil level, door open/close, air filter information, etc.) to the vehicle 1600.”, Para [0236]: “The deep-learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by the vehicle 1600 and, if the results do not match and the infrastructure concludes that the AI in the vehicle 1600 is malfunctioning, the server(s) 1678 may transmit a signal to the vehicle 1600 instructing a fail-safe computer of the vehicle 1600 to assume control, notify the passengers, and complete a safe parking maneuver.”).
Bae further teaches
determining a second driving scenario for the vehicle (Para [0408]: “For example, the processor 270 may determine the traveling state according to the driving environment, as the traveling state in the city road, the traveling state in the highway, parking situation, the curve traveling state, the slope traveling state, the traveling state in the backside road, the traveling state in the off-road, the traveling state in the snow road, the traveling state in the night, the traveling state in the traffic jam, and the like.”, Para [0415]: “As illustrated in FIG. 12B, when it is determined that the vehicle is traveling in a city road, the processor 270 may select AEB, LCA, HBA, LBA, BSD, and automatic parking as the traveling function”, Para [0416]: “When it is determined that the vehicle is traveling in a highway, the processor 270 may select AEB, ACC, LKA, TFA, HBA, LBA, BSD, and automatic parking as the traveling function.”).
Regarding claim 11, dependent upon claim 10, Smolyanskiy in view of Bae teaches everything regarding claim 10.
Smolyanskiy further discloses
modifying the input scale for the image data comprises at least one of: modifying the image data to a first image resolution based on the driving scenario, or modifying the image data to a first frame rate based on the driving scenario; and modifying the input scale for the second image data comprises at least one of: modifying the second image data to a second image resolution based on the second driving scenario, or modifying the second image data to a second frame rate based on the second driving scenario (Para [0048]: “Sensor detections may be accumulated over any desired window of time (e.g., 0.5 seconds (s), 1 s, 2 s, etc.). The size of the window may be selected based on the sensor and/or application (e.g., smaller windows may be selected for noisy applications such as highway scenarios).”; Due to the above 112b rejection, the “and” between the limitations of claim 9 is interpreted as “or”).
Claims 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Smolyanskiy et al. (US 2021/0150230 A1, hereinafter Smolyanskiy) in view of BAE et al. (US 2019/0276044 A1, hereinafter Bae) and YAO (CN 114170826 A, hereinafter Yao).
Regarding claim 15, dependent upon claim 8, Smolyanskiy in view of Bae teaches everything regarding claim 8.
Smolyanskiy further discloses
modifying input size for the image data based on the driving scenario, wherein the modified image data comprises the input scale modification and the input size modification (Para [0048]: “Sensor detections may be accumulated over any desired window of time (e.g., 0.5 seconds (s), 1 s, 2 s, etc.). The size of the window may be selected based on the sensor and/or application (e.g., smaller windows may be selected for noisy applications such as highway scenarios).”; Due to the above 112b rejection, the “and” between the limitations of claim 9 is interpreted as “or”).
However, Smolyanskiy in view of Bae does not explicitly teach
modifying input size for the image data based on the driving scenario, wherein the modified image data comprises the input scale modification and the input size modification.
Yao teaches
modifying input size for the image data based on the driving scenario, wherein the modified image data comprises the input scale modification and the input size modification (P. 6 Para. 1: “For example, in fig. 1, for example, when a vehicle needs to change lanes to the left, a left front camera needs to focus on target information of a short distance or a medium distance in the left rear direction, and a shooting range of the left front camera is large, an image area corresponding to a certain area range in the left rear direction of the vehicle in a left front camera shooting image can be used as a target image plane interesting area of the left front camera in the driving scene according to actual needs, so that resolution adjustment can be performed on image data corresponding to the target image plane interesting area, image quality is improved, and accuracy and reliability of a sensing result are improved.” P. 6 Para. 9 - P. 7 Para. 1: “the first image data may also be original image data acquired by the target image sensor, target image data corresponding to the target image plane region of interest is acquired from the original image data, and then resolution adjustment is performed on the target image data to acquire adjusted first image data, which may be specifically set according to actual requirements.”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Smolyanskiy in view of Bae and Yao with determining appropriate camera and region of interest within the camera image of Yao to effectively increase reliability of remote sensing.
Regarding claim 16, dependent upon claim 15, Smolyanskiy in view of Bae and Yao teaches everything regarding claim 15.
Yao further teaches
modifying the input size for the image data comprises at least one of: modifying a spatial input size for the image data based on the driving scenario; or modifying a temporal input size for the image data based on the driving scenario (P. 6 Para. 1: “For example, in fig. 1, for example, when a vehicle needs to change lanes to the left, a left front camera needs to focus on target information of a short distance or a medium distance in the left rear direction, and a shooting range of the left front camera is large, an image area corresponding to a certain area range in the left rear direction of the vehicle in a left front camera shooting image can be used as a target image plane interesting area of the left front camera in the driving scene according to actual needs, so that resolution adjustment can be performed on image data corresponding to the target image plane interesting area, image quality is improved, and accuracy and reliability of a sensing result are improved.” P. 6 Para. 9 - P. 7 Para. 1: “the first image data may also be original image data acquired by the target image sensor, target image data corresponding to the target image plane region of interest is acquired from the original image data, and then resolution adjustment is performed on the target image data to acquire adjusted first image data, which may be specifically set according to actual requirements.”).
Regarding claim 17, dependent upon claim 16, Smolyanskiy in view of Bae and Yao teaches everything regarding claim 16.
Yao further teaches
modifying the spatial input size for the image data comprises modifying a spatial region of interest size for the image data based on the driving scenario; and modifying the temporal input size for the image data comprises modifying a time duration for the image data based on the driving scenario (; Due to the above 112b rejection, the “and” between the limitations of claim 17 is interpreted as “or”.) (P. 6 Para. 1: “For example, in fig. 1, for example, when a vehicle needs to change lanes to the left, a left front camera needs to focus on target information of a short distance or a medium distance in the left rear direction, and a shooting range of the left front camera is large, an image area corresponding to a certain area range in the left rear direction of the vehicle in a left front camera shooting image can be used as a target image plane interesting area of the left front camera in the driving scene according to actual needs, so that resolution adjustment can be performed on image data corresponding to the target image plane interesting area, image quality is improved, and accuracy and reliability of a sensing result are improved.” P. 6 Para. 9 - P. 7 Para. 1: “the first image data may also be original image data acquired by the target image sensor, target image data corresponding to the target image plane region of interest is acquired from the original image data, and then resolution adjustment is performed on the target image data to acquire adjusted first image data, which may be specifically set according to actual requirements.”).
Allowable Subject Matter
Claims 1-6 and 18-20 are allowed.
Relevant Prior Art Directed to State of Art
Prakah-Asante et al. (US 9,988,037 B2, hereinafter Prakah-Asante) is prior art not applied in the rejection(s) above. Prakah-Asante discloses a vehicle processor for tracking vehicle input during vehicle traversal of a travel path within a location region, matching the vehicle input against a condition of a rule, such that when the vehicle input matches the condition, a driving scenario indication of a type corresponding to the rule is associated with the location region, and updating a location demand identifier associated with the location region based on the driving scenario indication.
Burger et al. (US 2023/0097950 A1, hereinafter Burger) is prior art not applied in the rejection(s) above. Burger discloses method for providing an environment image on the basis of a camera image of a vehicle camera of a vehicle for monitoring an environment of the vehicle.
Shen et al. (Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference, hereinafter Shen) is prior art not applied in the rejection(s) above. Shen discloses a Dynamic Fractional Skipping (DFS) framework. The core idea of DFS is to hypothesize layer-wise quantization (to different bitwidths) as intermediate “soft” choices to be made between fully utilizing and skipping a layer.
Yang et al. (Once For All Skip: Efficient Adaptive Deep Neural Networks, hereinafter Yang) is prior art not applied in the rejection(s) above. Yang discloses a new module, once for all skip (OFAS), for adaptive deep neural networks to efficiently control the block skip within a DNN model.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA CHEN whose telephone number is (703)756-5394. The examiner can normally be reached M-Th: 9:30 am - 4:30pm ET F: 9:30 am - 2:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, STEPHEN R KOZIOL can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J. C./ Examiner, Art Unit 2665
/Stephen R Koziol/ Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights