DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Final Office Action on the merits. Claims 1-20 are currently pending and are addressed below.
Response to Amendment
Claims 1-14 were rejected under 35 U.S.C. 101 as allegedly not directed to a statutory category of invention and claims 15-20 were rejected under 35 U.S.C. 101 as allegedly directed to non-statutory subject matter. Applicant amended claims 1, 8, and 15 accordingly; as such, the rejection is withdrawn.
Response to Arguments
Applicant’s arguments on pages 12-15 of the response, with respect to the rejection(s) of claim(s) 1-20 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Hantehzadeh and Hashimoto.
Claim Objections
Claims 2, 6-7, 9-10, 13-14, 16, 18, and 20 objected to because of the following informalities:
Claim 2 recites “…in response to the road issue being equal to or exceeding the threshold, the controlling of the vehicle to perform one or more of…”, in which the underlined portion appears to be grammatically incorrect.
Claims 2, 6-7, 9, 13-14, 16, 20 recites “…an other…”, in which the underlined portion appears to be a typographical error.
Claim 9 recites “…the processor is configured to determine comprises…”, in which the underlined portion appears to be grammatically incorrect.
Claim 10 recites “…configured to determining…”, which appears to be grammatically incorrect.
Claim 18 recites “…wherein the instructions cause the process to perform further comprising instructions for…”, which appears to be grammatically incorrect.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2 and 16 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 (and claim 16 by reciting analogous limitations) recites “…the predicted road issue…”. It is unclear if this is referring to the same road issue recited earlier in the claim: “…determining that the road issue…”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4, 6, 8-9, 11, 13, 15-16, 18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nayak of US 20230332911 A1, filed 04/19/2022, hereinafter “Nayak”, in view of Hantehzadeh of US 20210358105 A1, filed 05/14/2020, hereinafter “Hantehzadeh”, and further in view of Hashimoto of US 20190034762 A1, filed 04/23/2018, hereinafter “Hashimoto”.
Regarding claim 1, Nayak teaches:
A method performed by a processor of a vehicle, the method comprising: receiving, by a first neural network and a second neural network implemented by the processor, data from a sensor of the vehicle, as the vehicle travels on a road; (See at least [0053]: “The apparatus 10, such as the processing circuitry 12, may train any of a variety of machine learning models to identify road work project signs based upon a single or plurality of images. Examples of machine learning models that may be trained include…a neural network…In some example embodiments, the apparatus, such as the processing circuitry, is configured to separately train a plurality of different types of machine learning models utilizing the same training data including the same plurality of training examples.” See also [0022] regarding the system being on board the vehicle.)
generating, by the first neural network, a first output, and by the second neural network, a second output, (See at least [0053]: “…After having been trained, the apparatus, such as the processing circuitry, is configured to determine which of the plurality of machine learning models predicts vehicles based upon image data with the greatest accuracy. The machine learning model that has been identified as most accurate is thereafter utilized” & [0066]: “…The apparatus 10 identifies the road sign 64 via images from the camera system 22, traffic cameras, etc. and feeds those images into the machine learning model which determines the type of road sign present (e.g., a road construction sign, electronic sign, speed limit sign, etc.). The apparatus then takes this data along with relevant other information such as the image data of the actual road works/construction project 66 and feeds it back to the machine learning model (or to another model, algorithm, etc.) to determine if there is likely road construction on a given roadway.”)
predicting, by the AI model, a road issue associated with the road based on the first output and the second output; (See at least [0059-0060]: “Once trained, the machine learning model may then be provided various real-world data as mentioned in block 47 and used to determine roadworks locations based on the various data points above and others (block 48). A non-limiting example of the apparatus 10 detecting and/predicting a roadworks location is that of a car driving along a 2-lane highway. As the car drives down the road, it comes upon a sign which says, “Road Construction Ahead 2 Miles”. The apparatus 10 will capture images of the road sign via the car's camera system 22. The image data captured is provided to the machine learning model which, when trained, may identify the road sign as well as any relevant road construction objects in proximity to the roadway (e.g., construction vehicles, map objects, POIs, etc.).”)
determining an action to be performed by the vehicle based on the road issue; and controlling the vehicle to perform the action. (See at least [0063]: “The determination of the presence of a road construction can then be utilized in various ways…In some embodiments, the identified roadworks location may be used to activate autonomous or highly assisted driving features. For example, if the sedan discussed above had self-driving capabilities the apparatus 10 could activate the self-driving mode in response to the road construction to avoid congestion caused by it (and improve safety by avoiding construction workers).”)
Nayak does not explicitly teach:
wherein data input into the second neural network and the second output are modified using previous data input into the second neural network;
Hantehzadeh teaches:
wherein data input into the second neural network and the second output are modified using previous data input into the second neural network; (See at least [0069-0070]: “Recurrent Neural Networks may remember the past and RNN decisions may be influenced by what it has learned from the past. It should be noted that basic feed forward networks may “remember” things too, but those networks remember things they learned during training. For example, an image classifier may learn what a “1” looks like during training and then uses that knowledge to classify things in production. While RNNs learn similarly while training, in addition, RNNs may remember things learned from prior input(s) while generating output(s). Referring to FIG. 15, history may be part of the network. RNNs may take one or more input vectors 1501 and produce one or more output vectors 1502 and the output(s) 1502 may be influenced not just by weights applied on inputs like a regular NN, but also by a “hidden” state vector representing the context based on prior input(s)/output(s). The same input may produce a different output depending on previous inputs in the series.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak’s method with Hantehzadeh’s technique of modifying the data input and the second output for the second neural network using previous data input into the second neural network. Doing so would be obvious since “the success of CNNs and RNNs may be attributed to the concept of “parameter NNs” which may fundamentally be an effective way of leveraging the relationship between one input item and its surrounding neighbors in a more intrinsic fashion compared to a vanilla neural network” (See [0076] of Hantehzadeh).
Nayak and Hantehzadeh in combination do not explicitly teach:
extracting, by an artificial intelligence (Al) model implemented by the processor, features from the first output and the second output received from the first neural network and the second neural network;
Hashimoto teaches:
extracting, by an artificial intelligence (Al) model implemented by the processor, features from the first output and the second output received from the first neural network and the second neural network; (See at least Fig.1 & [0074]: “…a first neural network that receives first input data associated with an object, performs a common process associated with perception of the object based on the first input data, and outputs results of the common process; a second neural network that receives an output of the first neural network as second input data, performs a first perception process of perceiving the characteristics of the object with a first accuracy based on the second input data, and outputs results of the first perception process; and a third neural network that receives the output of the first neural network and intermediate data, which is generated by the second neural network in the course of the first perception process, as third input data, performs a second perception process of perceiving the characteristics of the object with a second accuracy which is higher than the first accuracy based on the third input data, and outputs results of the second perception process.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak and Hantehzadeh’s method with Hashimoto’s third neural network which receives outputs from a first and second neural network. Doing so would be obvious for “perceiving the characteristics of the object with a second accuracy which is higher than the first accuracy based on the third input data” (See [0074] of Hashimoto).
Regarding claim 2, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 1 as discussed above.
Nayak additionally teaches:
wherein determining the action to be performed comprises determining that the road issue equals or exceeds a threshold, and (See at least [0061]: “…since the car has passed a road sign indicating roadwork is occurring and there is construction equipment present, the apparatus 10 may predict that there is a high likelihood road construction is occurring” & [0069]: “In this example, there is a road construction project 66 obstructing the roadway but the road construction equipment, refuse, etc. may also be adjacent to the roadway 50 in some embodiments. The apparatus 10 uses all this information (e.g., data from the road sign, road construction equipment present, etc.) via the one or more machine learning models to determine if there is a road works project occurring. In this example, it is determined by the apparatus 10 that there is a high likelihood of such a project ahead of the sedan 60. This conclusion may be based on not only the presence of the road work equipment in the actual roadway but also ascertained by the apparatus from the road sign 64.”)
wherein the controlling of the vehicle to perform the action comprises: in response to the road issue being equal to or exceeding the threshold, the controlling of the vehicle to perform one or more of: maneuvering to avoid the road issue, sending a message to an other vehicle, proximate to the predicted road issue, wherein the message instructs the other vehicle to avoid the road issue, or sending a message to a server configured to process the message and direct an entity to remediate the road issue. (See at least [0063]: “…In some embodiments, the identified roadworks location may be used to activate autonomous or highly assisted driving features. For example, if the sedan discussed above had self-driving capabilities the apparatus 10 could activate the self-driving mode in response to the road construction to avoid congestion caused by it (and improve safety by avoiding construction workers).”)
Regarding claim 4, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 1 as discussed above.
Nayak additionally teaches:
comprising: extracting one or more features from the received data; and (See at least [0070]: “…the road sign 64 in this case might feature text which says “ROAD WORK AHEAD 500 FEET” along with being a traditional orange roadworks sign. The apparatus 10 can extract this text along with the shape and color of the sign to determine its meaning and content, at least in part, by comparing the current sign to a database of reference information.”)
verifying the road issue in response to the features. (See at least [0075]: “…if the road construction presence is confirmed by the presence of an actual road works project 66 on a given roadway (e.g., lanes blocked, equipment present, etc.) the confidence interval may be boosted up to 1 as the road sign's information has been confirmed as accurate. This confidence interval can be updated in real time and provide an actuate location for the road works project 66…”)
Regarding claim 6, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 1 as discussed above.
Nayak additionally teaches:
wherein the previous data is received from an other vehicle proximate to the road issue. (See at least [0057]: “For example, on a rural highway a road sign indicating roadwork is to occur from April-May of 2022. If the road sign is still present in August 2022 there is a distinct possibility that this road sign has been left in place by accident or that the road construction has been delayed past the stated period on the actual sign. As cars, trucks, etc. drive by this sign, the apparatus 10 may continually read and re-read the sign along with other various data points (time, date, if road work equipment is present, etc.) to confirm if the sign is accurate or not.”)
Regarding claim 8, Nayak teaches:
A vehicle comprising: a processor that executes instructions in a memory to configure the processor to: receive, by a first neural network and a second neural network implemented by the processor, data from a sensor of the vehicle, as the vehicle travels on a road; (See at least [0053]: “The apparatus 10, such as the processing circuitry 12, may train any of a variety of machine learning models to identify road work project signs based upon a single or plurality of images. Examples of machine learning models that may be trained include…a neural network…In some example embodiments, the apparatus, such as the processing circuitry, is configured to separately train a plurality of different types of machine learning models utilizing the same training data including the same plurality of training examples.” See also [0022] regarding the system being on board the vehicle.)
generate, by the first neural network, a first output, and by the second neural network, a second output, (See at least [0053]: “…After having been trained, the apparatus, such as the processing circuitry, is configured to determine which of the plurality of machine learning models predicts vehicles based upon image data with the greatest accuracy. The machine learning model that has been identified as most accurate is thereafter utilized” & [0066]: “…The apparatus 10 identifies the road sign 64 via images from the camera system 22, traffic cameras, etc. and feeds those images into the machine learning model which determines the type of road sign present (e.g., a road construction sign, electronic sign, speed limit sign, etc.). The apparatus then takes this data along with relevant other information such as the image data of the actual road works/construction project 66 and feeds it back to the machine learning model (or to another model, algorithm, etc.) to determine if there is likely road construction on a given roadway.”)
predict, by the Al model, a road issue associated with the road based on the first output and the second output; See at least [0059-0060]: “Once trained, the machine learning model may then be provided various real-world data as mentioned in block 47 and used to determine roadworks locations based on the various data points above and others (block 48). A non-limiting example of the apparatus 10 detecting and/predicting a roadworks location is that of a car driving along a 2-lane highway. As the car drives down the road, it comes upon a sign which says, “Road Construction Ahead 2 Miles”. The apparatus 10 will capture images of the road sign via the car's camera system 22. The image data captured is provided to the machine learning model which, when trained, may identify the road sign as well as any relevant road construction objects in proximity to the roadway (e.g., construction vehicles, map objects, POIs, etc.).”)
determine an action to be performed by the vehicle based on the road issue; and control the vehicle to perform the action. (See at least [0063]: “The determination of the presence of a road construction can then be utilized in various ways…In some embodiments, the identified roadworks location may be used to activate autonomous or highly assisted driving features. For example, if the sedan discussed above had self-driving capabilities the apparatus 10 could activate the self-driving mode in response to the road construction to avoid congestion caused by it (and improve safety by avoiding construction workers).”)
Nayak does not explicitly teach:
wherein data input into the second neural network and the second output are modified using previous data input into the second neural network;
Hantehzadeh teaches:
wherein data input into the second neural network and the second output are modified using previous data input into the second neural network; (See at least [0069-0070]: “Recurrent Neural Networks may remember the past and RNN decisions may be influenced by what it has learned from the past. It should be noted that basic feed forward networks may “remember” things too, but those networks remember things they learned during training. For example, an image classifier may learn what a “1” looks like during training and then uses that knowledge to classify things in production. While RNNs learn similarly while training, in addition, RNNs may remember things learned from prior input(s) while generating output(s). Referring to FIG. 15, history may be part of the network. RNNs may take one or more input vectors 1501 and produce one or more output vectors 1502 and the output(s) 1502 may be influenced not just by weights applied on inputs like a regular NN, but also by a “hidden” state vector representing the context based on prior input(s)/output(s). The same input may produce a different output depending on previous inputs in the series.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak’s method with Hantehzadeh’s technique of modifying the data input and the second output for the second neural network using previous data input into the second neural network. Doing so would be obvious since “the success of CNNs and RNNs may be attributed to the concept of “parameter NNs” which may fundamentally be an effective way of leveraging the relationship between one input item and its surrounding neighbors in a more intrinsic fashion compared to a vanilla neural network” (See [0076] of Hantehzadeh).
Nayak and Hantehzadeh in combination do not explicitly teach:
extract, by an artificial intelligence (AI) model implemented by the processor, features from the first output and the second output received from the first neural network and the second neural network;
Hashimoto teaches:
extract, by an artificial intelligence (AI) model implemented by the processor, features from the first output and the second output received from the first neural network and the second neural network; (See at least Fig.1 & [0074]: “…a first neural network that receives first input data associated with an object, performs a common process associated with perception of the object based on the first input data, and outputs results of the common process; a second neural network that receives an output of the first neural network as second input data, performs a first perception process of perceiving the characteristics of the object with a first accuracy based on the second input data, and outputs results of the first perception process; and a third neural network that receives the output of the first neural network and intermediate data, which is generated by the second neural network in the course of the first perception process, as third input data, performs a second perception process of perceiving the characteristics of the object with a second accuracy which is higher than the first accuracy based on the third input data, and outputs results of the second perception process.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak and Hantehzadeh’s method with Hashimoto’s third neural network which receives outputs from a first and second neural network. Doing so would be obvious for “perceiving the characteristics of the object with a second accuracy which is higher than the first accuracy based on the third input data” (See [0074] of Hashimoto).
Regarding claim 9, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 8 as discussed above.
Nayak additionally teaches:
wherein, when the processor determines the action to be performed, the processor is configured to determine comprises: determines that the road issue equals or exceeds a threshold, and (See at least [0061]: “…since the car has passed a road sign indicating roadwork is occurring and there is construction equipment present, the apparatus 10 may predict that there is a high likelihood road construction is occurring.”)
when the processor is configured to control the vehicle, the processor is configured to: in response to the road issue being equal to or exceeding the threshold, control the vehicle to perform one or more of: maneuver to avoid the road issue, sending a message to an other vehicle, proximate to the predicted road issue, wherein the message instructs the other vehicle to avoid the road issue, or send a message to a server configured to process the message and direct an entity to remediate the road issue. (See at least [0063]: “…In some embodiments, the identified roadworks location may be used to activate autonomous or highly assisted driving features. For example, if the sedan discussed above had self-driving capabilities the apparatus 10 could activate the self-driving mode in response to the road construction to avoid congestion caused by it (and improve safety by avoiding construction workers).”)
Regarding claim 11, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 8 as discussed above.
Nayak additionally teaches:
wherein the processor is configured to: extract one or more features from the received data; and (See at least [0070]: “…the road sign 64 in this case might feature text which says “ROAD WORK AHEAD 500 FEET” along with being a traditional orange roadworks sign. The apparatus 10 can extract this text along with the shape and color of the sign to determine its meaning and content, at least in part, by comparing the current sign to a database of reference information.”)
verify the road issue in response to the features. (See at least [0075]: “…if the road construction presence is confirmed by the presence of an actual road works project 66 on a given roadway (e.g., lanes blocked, equipment present, etc.) the confidence interval may be boosted up to 1 as the road sign's information has been confirmed as accurate. This confidence interval can be updated in real time and provide an actuate location for the road works project 66…”)
Regarding claim 13, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 8 as discussed above.
Nayak additionally teaches:
wherein the previous data is received from an other vehicle proximate to the road issue. (See at least [0057]: “For example, on a rural highway a road sign indicating roadwork is to occur from April-May of 2022. If the road sign is still present in August 2022 there is a distinct possibility that this road sign has been left in place by accident or that the road construction has been delayed past the stated period on the actual sign. As cars, trucks, etc. drive by this sign, the apparatus 10 may continually read and re-read the sign along with other various data points (time, date, if road work equipment is present, etc.) to confirm if the sign is accurate or not.”)
Regarding claim 15, Nayak teaches:
A non-transitory computer-readable medium comprising instructions that, when executed by a processor of a vehicle, cause the processor to perform: receiving, by a first neural network and a second neural network implemented by the processor, data from a sensor of the vehicle, as the vehicle travels on a road; (See at least [0053]: “The apparatus 10, such as the processing circuitry 12, may train any of a variety of machine learning models to identify road work project signs based upon a single or plurality of images. Examples of machine learning models that may be trained include…a neural network…In some example embodiments, the apparatus, such as the processing circuitry, is configured to separately train a plurality of different types of machine learning models utilizing the same training data including the same plurality of training examples.” See also [0022] regarding the system being on board the vehicle.)
generating, by the first neural network, a first output, and by the second neural network, a second output, (See at least [0053]: “…After having been trained, the apparatus, such as the processing circuitry, is configured to determine which of the plurality of machine learning models predicts vehicles based upon image data with the greatest accuracy. The machine learning model that has been identified as most accurate is thereafter utilized” & [0066]: “…The apparatus 10 identifies the road sign 64 via images from the camera system 22, traffic cameras, etc. and feeds those images into the machine learning model which determines the type of road sign present (e.g., a road construction sign, electronic sign, speed limit sign, etc.). The apparatus then takes this data along with relevant other information such as the image data of the actual road works/construction project 66 and feeds it back to the machine learning model (or to another model, algorithm, etc.) to determine if there is likely road construction on a given roadway.”)
predicting, by the Al model, a road issue associated with the road based on the first output and the second output; (See at least [0059-0060]: “Once trained, the machine learning model may then be provided various real-world data as mentioned in block 47 and used to determine roadworks locations based on the various data points above and others (block 48). A non-limiting example of the apparatus 10 detecting and/predicting a roadworks location is that of a car driving along a 2-lane highway. As the car drives down the road, it comes upon a sign which says, “Road Construction Ahead 2 Miles”. The apparatus 10 will capture images of the road sign via the car's camera system 22. The image data captured is provided to the machine learning model which, when trained, may identify the road sign as well as any relevant road construction objects in proximity to the roadway (e.g., construction vehicles, map objects, POIs, etc.).”)
determining an action to be performed by the vehicle based on the road issue; and controlling the vehicle to perform the action. (See at least [0063]: “The determination of the presence of a road construction can then be utilized in various ways…In some embodiments, the identified roadworks location may be used to activate autonomous or highly assisted driving features. For example, if the sedan discussed above had self-driving capabilities the apparatus 10 could activate the self-driving mode in response to the road construction to avoid congestion caused by it (and improve safety by avoiding construction workers).”)
Nayak does not explicitly teach:
wherein data input into the second neural network and the second output are modified using previous data input into the second neural network;
Hantehzadeh teaches:
wherein data input into the second neural network and the second output are modified using previous data input into the second neural network; (See at least [0069-0070]: “Recurrent Neural Networks may remember the past and RNN decisions may be influenced by what it has learned from the past. It should be noted that basic feed forward networks may “remember” things too, but those networks remember things they learned during training. For example, an image classifier may learn what a “1” looks like during training and then uses that knowledge to classify things in production. While RNNs learn similarly while training, in addition, RNNs may remember things learned from prior input(s) while generating output(s). Referring to FIG. 15, history may be part of the network. RNNs may take one or more input vectors 1501 and produce one or more output vectors 1502 and the output(s) 1502 may be influenced not just by weights applied on inputs like a regular NN, but also by a “hidden” state vector representing the context based on prior input(s)/output(s). The same input may produce a different output depending on previous inputs in the series.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak’s method with Hantehzadeh’s technique of modifying the data input and the second output for the second neural network using previous data input into the second neural network. Doing so would be obvious since “the success of CNNs and RNNs may be attributed to the concept of “parameter NNs” which may fundamentally be an effective way of leveraging the relationship between one input item and its surrounding neighbors in a more intrinsic fashion compared to a vanilla neural network” (See [0076] of Hantehzadeh).
Nayak and Hantehzadeh in combination do not explicitly teach:
extracting, by an artificial intelligence (AI) model implemented by the processor, features from the first output and the second output received from the first neural network and the second neural network;
Hashimoto teaches:
extracting, by an artificial intelligence (Al) model implemented by the processor, features from the first output and the second output received from the first neural network and the second neural network; (See at least Fig.1 & [0074]: “…a first neural network that receives first input data associated with an object, performs a common process associated with perception of the object based on the first input data, and outputs results of the common process; a second neural network that receives an output of the first neural network as second input data, performs a first perception process of perceiving the characteristics of the object with a first accuracy based on the second input data, and outputs results of the first perception process; and a third neural network that receives the output of the first neural network and intermediate data, which is generated by the second neural network in the course of the first perception process, as third input data, performs a second perception process of perceiving the characteristics of the object with a second accuracy which is higher than the first accuracy based on the third input data, and outputs results of the second perception process.”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak and Hantehzadeh’s method with Hashimoto’s third neural network which receives outputs from a first and second neural network. Doing so would be obvious for “perceiving the characteristics of the object with a second accuracy which is higher than the first accuracy based on the third input data” (See [0074] of Hashimoto).
Regarding claim 16, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 15 as discussed above.
Nayak additionally teaches:
wherein determining the action to be performed comprises determining that the road issue equals or exceeds a threshold, and (See at least [0061]: “…since the car has passed a road sign indicating roadwork is occurring and there is construction equipment present, the apparatus 10 may predict that there is a high likelihood road construction is occurring.”)
wherein the controlling of the vehicle to perform the action comprises: in response to the road issue being equal to or exceeding the threshold, controlling the vehicle to perform one or more of: maneuvering to avoid the road issue, sending a message to an other vehicle, proximate to the predicted road issue, wherein the message instructs the other vehicle to avoid the road issue, or sending a message to an other vehicle, proximate to the predicted road issue, wherein the message instructs the other vehicle to avoid the road issue, or sending a message to a server configured to process the message and direct an entity to remediate the road issue. (See at least [0063]: “…In some embodiments, the identified roadworks location may be used to activate autonomous or highly assisted driving features. For example, if the sedan discussed above had self-driving capabilities the apparatus 10 could activate the self-driving mode in response to the road construction to avoid congestion caused by it (and improve safety by avoiding construction workers).”)
Regarding claim 18, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 15 as discussed above.
Nayak additionally teaches:
wherein the instructions cause the process to perform further comprising instructions for: extracting one or more features from the received data; and (See at least [0070]: “…the road sign 64 in this case might feature text which says “ROAD WORK AHEAD 500 FEET” along with being a traditional orange roadworks sign. The apparatus 10 can extract this text along with the shape and color of the sign to determine its meaning and content, at least in part, by comparing the current sign to a database of reference information.”)
verifying the road issue in response to the extracted features.(See at least [0075]: “…if the road construction presence is confirmed by the presence of an actual road works project 66 on a given roadway (e.g., lanes blocked, equipment present, etc.) the confidence interval may be boosted up to 1 as the road sign's information has been confirmed as accurate. This confidence interval can be updated in real time and provide an actuate location for the road works project 66…”)
Regarding claim 20, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 15 as discussed above.
Nayak additionally teaches:
wherein the previous data is received from an other vehicle proximate to the road issue. (See at least [0057]: “For example, on a rural highway a road sign indicating roadwork is to occur from April-May of 2022. If the road sign is still present in August 2022 there is a distinct possibility that this road sign has been left in place by accident or that the road construction has been delayed past the stated period on the actual sign. As cars, trucks, etc. drive by this sign, the apparatus 10 may continually read and re-read the sign along with other various data points (time, date, if road work equipment is present, etc.) to confirm if the sign is accurate or not.”)
Claim(s) 3, 10, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nayak in view of Hantehzadeh and Hashimoto and further in view of Shao of CN 119091332 A, filed 09/11/2024, hereinafter “Shao ‘332”.
Regarding claim 3, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 1 as discussed above.
Nayak additionally teaches:
transmitting a message identifying the road issue to an entity before the latest time. (See at least [0063]: “The determination of the presence of a road construction can then be utilized in various ways. The apparatus 10 may alert the driver of the sedan (and others) via graphical user interface that there could be a delay or risk ahead…”)
Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach:
comprising determining a latest time to remediate the road issue, before a repair becomes necessary; and
However, Nayak teaches determining that construction “has been delayed past the stated period on the actual sign” based on the sign still being present on the road after the stated period (See at least [0057]). Additionally, Shao ‘332 teaches determining that a road defect needs immediate repair if the hazard level exceeds a preset threshold (See at least [0112] & [0114]) and issuing “timely warnings” for road defects (See at least [0048] & [0125]). Therefore, the combination of Nayak, Hantehzadeh, Hashimoto, and Shao ‘332 renders obvious determining a latest time to remediate a road issue before repair becomes necessary, which provides the benefit of “timely defect detection and accurate hazard assessment help to promptly identify and repair potential safety hazards, thereby improving the overall road safety and driving comfort” (See [0109] of Shao ‘332).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak, Hantehzadeh, and Hashimoto’s method with Shao ‘332’s technique of determining a latest time to remediate a road issue before repair becomes necessary and transmitting a message identifying the road issue to an entity prior to the latest time. Doing so would be obvious since “timely defect detection and accurate hazard assessment help to promptly identify and repair potential safety hazards, thereby improving the overall road safety and driving comfort” (See [0109] of Shao ‘332).
Regarding claim 10, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 8 as discussed above.
Nayak additionally teaches:
transmit a message that identifies the road issue to an entity before the latest time. (See at least [0063]: “The determination of the presence of a road construction can then be utilized in various ways. The apparatus 10 may alert the driver of the sedan (and others) via graphical user interface that there could be a delay or risk ahead…”)
Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach:
wherein the processor is configured to determining a latest time to remediate the road issue, before a repair becomes necessary; and
However, Nayak teaches determining that construction “has been delayed past the stated period on the actual sign” based on the sign still being present on the road after the stated period (See at least [0057]). Additionally, Shao ‘332 teaches determining that a road defect needs immediate repair if the hazard level exceeds a preset threshold (See at least [0112] & [0114]) and issuing “timely warnings” for road defects (See at least [0048] & [0125]). Therefore, the combination of Nayak, Hantehzadeh, Hashimoto and Shao ‘332 renders obvious determining a maximum delay before a repair becomes necessary, which provides the benefit of “timely defect detection and accurate hazard assessment help to promptly identify and repair potential safety hazards, thereby improving the overall road safety and driving comfort” (See [0109] of Shao ‘332).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak, Hantehzadeh, and Hashimoto’s method with Shao ‘332’s technique of determining a latest time to remediate a road issue before repair becomes necessary and transmitting a message identifying the road issue to an entity prior to the latest time. Doing so would be obvious since “timely defect detection and accurate hazard assessment help to promptly identify and repair potential safety hazards, thereby improving the overall road safety and driving comfort” (See [0109] of Shao ‘332).
Regarding claim 17, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 15 as discussed above.
Nayak additionally teaches:
transmitting a message identifying the road issue to an entity prior before the latest time. (See at least [0063]: “The determination of the presence of a road construction can then be utilized in various ways. The apparatus 10 may alert the driver of the sedan (and others) via graphical user interface that there could be a delay or risk ahead…”)
Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach:
wherein the instructions cause the process to perform: determining a latest time to remediate the road issue, before a repair becomes necessary; and
However, Nayak teaches determining that construction “has been delayed past the stated period on the actual sign” based on the sign still being present on the road after the stated period (See at least [0057]). Additionally, Shao ‘332 teaches determining that a road defect needs immediate repair if the hazard level exceeds a preset threshold (See at least [0112] & [0114]) and issuing “timely warnings” for road defects (See at least [0048] & [0125]). Therefore, the combination of Nayak, Hantehzadeh, Hashimoto, and Shao ‘332 renders obvious determining a latest time to remediate a road issue before repair becomes necessary, which provides the benefit of “timely defect detection and accurate hazard assessment help to promptly identify and repair potential safety hazards, thereby improving the overall road safety and driving comfort” (See [0109] of Shao ‘332).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak, Hantehzadeh, and Hashimoto’s method with Shao ‘332’s technique of determining a maximum delay before a repair becomes necessary and transmitting a message identifying the road issue to an entity prior to the latest time. Doing so would be obvious since “timely defect detection and accurate hazard assessment help to promptly identify and repair potential safety hazards, thereby improving the overall road safety and driving comfort” (See [0109] of Shao ‘332).
Claim(s) 5, 12, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nayak in view of Hantehzadeh and Hashimoto and further in view of Shao of CN 118609372 A, filed 08/07/2024, hereinafter “Shao ‘372”.
Regarding claim 5, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 1 as discussed above.
Nayak additionally teaches:
wherein the first neural network (See at least [0066]: “The apparatus 10 identifies the road sign 64 via images from the camera system 22, traffic cameras, etc. and feeds those images into the machine learning model which determines the type of road sign present (e.g., a road construction sign, electronic sign, speed limit sign, etc.). The apparatus then takes this data along with relevant other information such as the image data of the actual road works/construction project 66 and feeds it back to the machine learning model (or to another model, algorithm, etc.) to determine if there is likely road construction on a given roadway.”)
wherein, the predicting of the road issue comprises: mapping, (See at least [0062]: “The machine learning model in this example makes its determination based on a combination of specific factors (map data, image data, etc.), and the model predicts the presence of roadworks because of specific factors in a specific combination or configuration are present. The factors in this example may include the image data of the road sign and dump truck (together in one image or as separate images), image data of the roadways, image data of objects proximate to the roadway (e.g., dumpsters, construction materials, construction workers, etc.) as well as time of day data, historic data, etc. This set of data, provided to the model, matches (or is like) the factors used in the training process (in this example). This allows the machine learning model to predict a roadworks location given the location, time of day, vehicles, and animate objects present, etc.”)
Hantehzadeh additionally teaches that the first neural network is a CNN that processes images and the second neural network is an RNN that processes “each CNN-specific feature included in the respective one or more CNN-specific features of the each image determined by the respective CNN” (See at least Fig. 1, [0040] & [0065]).
However, Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach that the mapping of features to the road issue is performed by an AI model.
However, Shao ‘372 teaches a neural network that processes time series data to predict “dynamically changing road condition information,” where “effective traffic features” are extracted to perform “real-time traffic calculation and prediction” (See at least [n0025] & [0120-0123]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak, Hantehzadeh, and Hashimoto’s method with Shao ‘372’s technique of using an neural network to map features to a road issue. Doing so would be obvious for “providing accurate prediction results for traffic management and decision-making” (See [n0062] of Shao ‘372).
Regarding claim 12, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 8 as discussed above.
Nayak additionally teaches:
wherein the first neural network (See at least [0066]: “The apparatus 10 identifies the road sign 64 via images from the camera system 22, traffic cameras, etc. and feeds those images into the machine learning model which determines the type of road sign present (e.g., a road construction sign, electronic sign, speed limit sign, etc.). The apparatus then takes this data along with relevant other information such as the image data of the actual road works/construction project 66 and feeds it back to the machine learning model (or to another model, algorithm, etc.) to determine if there is likely road construction on a given roadway.”)
wherein, when the processor is configured to predict the road issue, the processor is configured to: map, (See at least [0062]: “The machine learning model in this example makes its determination based on a combination of specific factors (map data, image data, etc.), and the model predicts the presence of roadworks because of specific factors in a specific combination or configuration are present. The factors in this example may include the image data of the road sign and dump truck (together in one image or as separate images), image data of the roadways, image data of objects proximate to the roadway (e.g., dumpsters, construction materials, construction workers, etc.) as well as time of day data, historic data, etc. This set of data, provided to the model, matches (or is like) the factors used in the training process (in this example). This allows the machine learning model to predict a roadworks location given the location, time of day, vehicles, and animate objects present, etc.”)
Hantehzadeh additionally teaches that the first neural network is a CNN that processes images and the second neural network is an RNN that processes “each CNN-specific feature included in the respective one or more CNN-specific features of the each image determined by the respective CNN” (See at least Fig. 1, [0040] & [0065]).
However, Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach that the mapping of features to the road issue is performed by an AI model.
However, Shao ‘372 teaches a neural network that processes time series data to predict “dynamically changing road condition information,” where “effective traffic features” are extracted to perform “real-time traffic calculation and prediction” (See at least [n0025] & [0120-0123]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak, Hantehzadeh, and Hashimoto’s method with Shao ‘372’s technique of using an neural network to map features to a road issue. Doing so would be obvious for “providing accurate prediction results for traffic management and decision-making” (See [n0062] of Shao ‘372).
Regarding claim 19, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 15 as discussed above.
Nayak additionally teaches:
wherein the first neural network (See at least [0066]: “The apparatus 10 identifies the road sign 64 via images from the camera system 22, traffic cameras, etc. and feeds those images into the machine learning model which determines the type of road sign present (e.g., a road construction sign, electronic sign, speed limit sign, etc.). The apparatus then takes this data along with relevant other information such as the image data of the actual road works/construction project 66 and feeds it back to the machine learning model (or to another model, algorithm, etc.) to determine if there is likely road construction on a given roadway.”)
wherein, the predicting of the road issue comprises: mapping, (See at least [0062]: “The machine learning model in this example makes its determination based on a combination of specific factors (map data, image data, etc.), and the model predicts the presence of roadworks because of specific factors in a specific combination or configuration are present. The factors in this example may include the image data of the road sign and dump truck (together in one image or as separate images), image data of the roadways, image data of objects proximate to the roadway (e.g., dumpsters, construction materials, construction workers, etc.) as well as time of day data, historic data, etc. This set of data, provided to the model, matches (or is like) the factors used in the training process (in this example). This allows the machine learning model to predict a roadworks location given the location, time of day, vehicles, and animate objects present, etc.”)
Hantehzadeh additionally teaches that the first neural network is a CNN that processes images and the second neural network is an RNN that processes “each CNN-specific feature included in the respective one or more CNN-specific features of the each image determined by the respective CNN” (See at least Fig. 1, [0040] & [0065]).
However, Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach that the mapping of features to the road issue is performed by an AI model.
However, Shao ‘372 teaches a neural network that processes time series data to predict “dynamically changing road condition information,” where “effective traffic features” are extracted to perform “real-time traffic calculation and prediction” (See at least [n0025] & [0120-0123]).
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak, Hantehzadeh, and Hashimoto’s method with Shao ‘372’s technique of using an neural network to map features to a road issue. Doing so would be obvious for “providing accurate prediction results for traffic management and decision-making” (See [n0062] of Shao ‘372).
Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nayak in view of Hantehzadeh and Hashimoto and further in view of Bai of US 10166991 B1, filed 12/01/2017, hereinafter “Bai”.
Regarding claim 7, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 1 as discussed above.
Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach:
wherein the previous data is weighted based on a reputation score of an other vehicle that provided the previous data.
Bai teaches:
wherein the previous data is weighted based on a reputation score of an other vehicle that provided the previous data. (See at least cols. 11-12, lines 49-67 & 1-9: “The example decision module 710 is configured to determine whether or not the vehicle should transmit event information to the cloud-based server. In this example, the example decision module 710 determines that the vehicle should transmit event information if three conditions exists…The third condition is that the reputation score (R.sub.k) is above a third threshold level (δ(0).”)
One having ordinary skill in the art, before the effective filing date of the claimed invention, would have found it obvious to combine Nayak, Hantehzadeh, and Hashimoto’s method with Bai’s previous data being weighted based on a reputation score of another vehicle that provided the previous data. Doing so would be obvious to ensure the vehicle is “providing accurate event observation information” (See col. 12, line 37 of Bai).
Regarding claim 14, Nayak, Hantehzadeh, and Hashimoto in combination teach all the limitations of claim 8 as discussed above.
Nayak, Hantehzadeh, and Hashimoto in combination do not explicitly teach:
wherein the previous data is weighted based on a reputation score of an other vehicle that provided the previous data.
Bai teaches:
wherein the previous data is weighted based on a reputation score of an other vehicle that provided the previous data. (See at least cols. 11-12, lines 49-67 & 1-9: “The example decision module 710 is configured to determine whether or not the vehicle should transmit event information to the cloud-based server. In this example, the example decision module 710 determines that the vehicle should transmit event information if three conditions exists…The third condition is that the reputation score (R.sub.k) is above a third threshold level (δ(0).”)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NIKKI MARIE M MOLINA whose telephone number is (571)272-5180. The examiner can normally be reached M-F, 9am-6pm PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aniss Chad can be reached at 571-270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NIKKI MARIE M MOLINA/Examiner, Art Unit 3662
/ANISS CHAD/Supervisory Patent Examiner, Art Unit 3662
Prosecution Insights