DETAILED ACTION
Claims 1-20 are currently pending and have been examined in this application.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the “request for continued examination” filed 07/09/2026.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5-6, 9-12, 15-17, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu (US20240062555) in view of Wang (US11767024) further in view of Bhattacharyya (US20250119625).
Claim 1:
Xu Explicitly teaches:
One or more processors comprising: processing circuity
(Xu) – “The road sign interpretation system 10 utilizes an on-board computer 26 which is programmed to operate and perform a method of operating the road sign interpretation system 10. The on-board computer 26 receives the image data 16 noted above from the camera 12, as well as other data such as data saved and retrieved from a memory 28 to perform road sign interpretation and to generate and display data results. The on-board computer 26 described in reference to FIG. 1 is programmed to perform operations related to a system and method to collect image data and interpret road signs for the road sign interpretation system 10. The on-board computer 26 is a non-generalized, electronic control device having a preprogrammed digital controller or processor, the memory 28 or a similar non-transitory computer readable medium used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver or input/output ports. The computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.” (Para 0035)
receive image data representing one or more objects in an environment, the image data generated using one or more sensors of an ego-machine traversing the environment;
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
extract one or more first natural language characters represented in the image data;
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
provide a representation of the one or more first natural language characters that were extracted from the image data as [at least a portion of one or more text prompts [[an]] input [[in]]to one or more language
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
Examiner Note: Bracketed text not explicitly taught by primary reference, but is taught by non-primary reference later in the rejection.
cause presentation, using at least one of a display device or a sound device associated with an operator or occupant of the ego-machine, of a representation of the [one or more second natural language characters that were generated by the one or more language models].
(Xu) – “In another aspect of the present disclosure, an on-board computer receives the image data from the camera, in addition to data saved and retrieved from a memory of the on-board computer to perform road sign interpretation and to generate and display data results.” (Para 0007)
“The camera 12 may be any type of color camera and is forward or front-facing to receive a data set having image data 16 including data defining a roadway 18 upon which the vehicle 14 is traveling on and toward, data of one or more other vehicles 20 also on the roadway 18, and data of at least one road sign 22 providing information to an occupant or operator of the vehicle 14 related to the roadway 18. According to several aspects, the data set of the image data 16 received by the camera 12 may be live displayed on a screen 24 of the camera 12 or displayed in a similar display provided in the vehicle 14.” (Para 0034)
Examiner Note: Bracketed text not explicitly taught by primary reference, but is taught by non-primary reference later in the rejection.
Xu does not explicitly teach:
at least a portion of one or more text prompts [[an]] input [[in]]to one or more language that were generated by the one or more language models
Wang, in the same field of endeavor of image analysis, teaches:
to generate one or more second natural language characters… one or more second natural language characters
(Wang) – “Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
PNG
media_image1.png
350
368
media_image1.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the apparatus for driving assistance of Wang. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “the AR technology can be applied in the vehicle travelling process to assist a driver in better mastering driving information in the vehicle travelling process, and the user experience can be improved.” (Wang Abstract)
Wang does not explicitly teach:
at least a portion of one or more text prompts input to one or more language models … that were generated by the one or more language models
Bhattacharyya, in the same field of endeavor of prompt generation, teaches:
at least a portion of one or more text prompts input to one or more language models … that were generated by the one or more language models
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Claim 2:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu further teaches:
wherein the one or more objects include a traffic sign, and wherein the processing circuitry is further to: determine, via object detection and based on the image data, that the one or more regions of the image data depict at least a portion of the traffic sign;
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
extract the one or more first natural language characters represented in the traffic sign based at least on performing optical character recognition (OCR) within the one or more regions of the image data in response to determining that the one or more regions of the image data depict at least a portion of the traffic sign ; and
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
Xu does not explicitly teach:
generate the one or more text prompts comprising the representation of the one or more first natural language characters extracted from the one or more regions of the image data depicting at least the portion of the traffic sign.
Bhattacharyya, in the same field of endeavor of prompt generation, teaches:
generate the one or more text prompts comprising the representation of the one or more first natural language characters extracted from the one or more regions of the image data depicting at least the portion of the traffic sign.
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Claim 5:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu further teaches:
wherein the one or more objects include a parking sign that includes parking instructions corresponding to at least one parking spot, and wherein the processing circuitry is further to: provide a representation of parking context as at least a second portion of [the one or more text prompts input [[in]]to the one or more language
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“In a fourth step (4) individual sign instances 46 from the first CNN 44 and individual ones of the text instances 42 in the digitized forms 56 from the OCR 52 are together fed into a sign text synthesizer module 60. The sign text synthesizer module 60 evaluates a text-sign membership including whether or not sign text lies within a bounding region of one of the sign instances 46 such as a sign bounding box 62. The sign text synthesizer module 60 also rearranges and configures the detected sign text into a logical reading order, including for example left-to-right and top-to-bottom as individual text instances or as synthesized text instances 64. For example, “Physician Parking” and “Doctors only” consist of entirely different strings, but both refer to the same parking guideline/instruction of sign category. To address this challenge, a text processing component is developed that enables semantic understanding of the detected sign text.” (Para 0040)
“In a fifth step (5) the text instances 42 are fed into a semantic encoding and interpretation module 68 to identify high-level semantics of the detected road signs. Specifically, individual ones of the text instances 42 are first encoded as fixed-dimension feature vectors 70 using the semantic encoding and interpretation module 68. Subsequently, individual sign categories 72 are automatically classified as one of the possible road sign categories, for example “no parking” or “detour”, based on the semantics captured as the feature vectors 70.” (Para 0043)
Xu does not explicitly teach:
the one or more text prompts input [[in]]to the one or more language generated by the one or more language models includes a natural language phrase indicating a prediction of whether the at least one parking spot is permitted to be occupied by the ego-machine based at least on the parking context and the parking instructions.
Wang, in the same field of endeavor of image analysis, teaches:
wherein the parking context includes at least one of: a time of day that the elements representing the parking sign was detected, a day of week that the elements representing the parking sign was detected, an ego-machine type of the ego-machine, or an indication of whether the at least one parking spot is available for parking, and
(Wang) – “Further, the step of displaying virtual 3D display information corresponding to the driving assistance information is implemented by at least one of the following modes: …when parking area information is to be displayed, displaying virtual 3D display information corresponding to at least one of areas where parking is allowed and it is appropriate for parking, areas where parking is allowed but it is not appropriate for parking, and areas where parking is not allowed.” (Col 7 Ln 36-64)
“when parking area information is to be displayed, according to at least one of the presence of a NO PARKING sign or like in the surrounding area of the current vehicle, the size of the current vehicle and the current road surface condition, determining at least one of areas where parking is allowed and it is appropriate for parking, areas where parking is allowed but it is not appropriate for parking, and areas where parking is not allowed.” (Col 8 Ln 37-44)
“Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
wherein the presentation of the representation of one or more second natural language characters …includes a natural language phrase indicating a prediction of whether the at least one parking spot is permitted to be occupied by the ego-machine based at least on the parking context and the parking instructions.
(Wang) – “Further, the step of displaying virtual 3D display information corresponding to the driving assistance information is implemented by at least one of the following modes: …when parking area information is to be displayed, displaying virtual 3D display information corresponding to at least one of areas where parking is allowed and it is appropriate for parking, areas where parking is allowed but it is not appropriate for parking, and areas where parking is not allowed.” (Col 7 Ln 36-64)
“when parking area information is to be displayed, according to at least one of the presence of a NO PARKING sign or like in the surrounding area of the current vehicle, the size of the current vehicle and the current road surface condition, determining at least one of areas where parking is allowed and it is appropriate for parking, areas where parking is allowed but it is not appropriate for parking, and areas where parking is not allowed.” (Col 8 Ln 37-44)
“Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the apparatus for driving assistance of Wang. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “the AR technology can be applied in the vehicle travelling process to assist a driver in better mastering driving information in the vehicle travelling process, and the user experience can be improved.” (Wang Abstract)
Wang does not explicitly teach:
the one or more text prompts input [[in]]to the one or more language generated by the one or more language models
Bhattacharyya, in the same field of endeavor of prompt generation, teaches:
the one or more text prompts input [[in]]to the one or more language generated by the one or more language models
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Claim 6:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu further teaches:
provide the at least one of: [the weather data, the road condition data, the traffic data, the event data, or the geo-location indicator] as at least a second portion of the [one or more text prompts input [[in]]to the one or more language
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
Examiner Note: Bracketed text not explicitly taught by primary reference, but is taught by non-primary reference later in the rejection.
Xu does not explicitly teach:
wherein the processing circuitry is further to: receive a geo-location indicator that represents a location of the ego-machine in the environment; based at least on the geo-location indicator, determine at least one of: weather data, road condition data, traffic data, or event data associated with the geo-location; and… the weather data, the road condition data, the traffic data, the event data, or the geo-location indicator … one or more text prompts input [[in]]to the one or more language
Wang, in the same field of endeavor of image analysis, teaches:
wherein the one or more processing units are further to: receive a geo-location indicator that represents a location of the ego-machine in the environment;
(Wang) – “With regard to the sensing of the road environment, the apparatus can sense the road environment by at least one of the following modes: sensing by at least one sensor borne on the apparatus; sensing by at least sensor borne on the vehicle; acquiring information from at least one of the following devices in a communication manner: an apparatus of the same type, an apparatus of a different type and other vehicles; and, acquiring information by a Global Positioning System (GPS).” (Col 5 Ln 4-12)
“Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
based at least on the geo-location indicator, determine at least one of: weather data, road condition data, traffic data, or event data associated with the geo-location; and
(Wang) – “With regard to the sensing of the road environment, the apparatus can sense the road environment by at least one of the following modes: sensing by at least one sensor borne on the apparatus; sensing by at least sensor borne on the vehicle; acquiring information from at least one of the following devices in a communication manner: an apparatus of the same type, an apparatus of a different type and other vehicles; and, acquiring information by a Global Positioning System (GPS).” (Col 5 Ln 4-12)
“Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
“The target driving assistance information can comprise, but not be limited to, prompt information related to the traffic safety, driving environment, information indication of traffic and road condition or more, traffic regulations, in-vehicle information, or more. Specifically, a target object related to the target driving assistance information during the driving process can comprise, but not be limited to, lane lines, lane dividing rails, lanes, surrounding motor vehicles, surrounding non-motor vehicles, surrounding pedestrians, surrounding trees, surrounding buildings, road surface ruts, information indication of traffic and road condition or more, traffic policemen, objects within a blind area of a side-view mirror, objects on a rear seat in this vehicle, an external area of the trail, the dashboard, or more.” (Col 11 Ln 33-47)
the weather data, the road condition data, the traffic data, the event data, or the geo-location indicator … wherein the presentation, using the display device or the sound device associated with the operator of the ego-machine, of the representation of the one or more second natural language characters includes a natural language phrase representing a summary of the at least one of: the weather data, the road condition data, the traffic data, or the event data.
(Wang) – “With regard to the sensing of the road environment, the apparatus can sense the road environment by at least one of the following modes: sensing by at least one sensor borne on the apparatus; sensing by at least sensor borne on the vehicle; acquiring information from at least one of the following devices in a communication manner: an apparatus of the same type, an apparatus of a different type and other vehicles; and, acquiring information by a Global Positioning System (GPS).” (Col 5 Ln 4-12)
“Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
“The target driving assistance information can comprise, but not be limited to, prompt information related to the traffic safety, driving environment, information indication of traffic and road condition or more, traffic regulations, in-vehicle information, or more. Specifically, a target object related to the target driving assistance information during the driving process can comprise, but not be limited to, lane lines, lane dividing rails, lanes, surrounding motor vehicles, surrounding non-motor vehicles, surrounding pedestrians, surrounding trees, surrounding buildings, road surface ruts, information indication of traffic and road condition or more, traffic policemen, objects within a blind area of a side-view mirror, objects on a rear seat in this vehicle, an external area of the trail, the dashboard, or more.” (Col 11 Ln 33-47)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the apparatus for driving assistance of Wang. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “the AR technology can be applied in the vehicle travelling process to assist a driver in better mastering driving information in the vehicle travelling process, and the user experience can be improved.” (Wang Abstract)
Wang does not explicitly teach:
one or more text prompts [[an]] input [[in]]to one or more language
Bhattacharyya, in the same field of endeavor of text prompt generation, teaches:
one or more text prompts [[an]] input [[in]]to one or more language
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Claim 9:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu further teaches:
wherein the processing circuitry is further to: access, from one or more data sources, destination or travel route information associated with a destination or a travel route of the ego-machine; and
(Xu) – “In another aspect of the present disclosure, a planner and router module receives an output of the semantic encoding and interpretation module. A route plan is computed a planner portion which is received by a navigation router portion.” (Para 0006)
“In another aspect of the present disclosure, the method includes: forwarding an output of the semantic encoding and interpretation module to a planner; and operating the planner to compute a route plan and feeding the route plan to a navigation router.” (Para 0015)
“In another aspect of the present disclosure, the method includes: recommending a route for the vehicle to take that optimizes travel time and distance for navigation using the navigation router and navigates the vehicle in case of an autonomous vehicle or communicates the route to a vehicle operator.” (Para 0016)
provide a representation of the destination or travel route information as at least a second portion of the [one or more text prompts input [[in]]to the one or more language
(Xu) – “In another aspect of the present disclosure, a planner and router module receives an output of the semantic encoding and interpretation module. A route plan is computed a planner portion which is received by a navigation router portion.” (Para 0006)
“In another aspect of the present disclosure, the method includes: forwarding an output of the semantic encoding and interpretation module to a planner; and operating the planner to compute a route plan and feeding the route plan to a navigation router.” (Para 0015)
“In another aspect of the present disclosure, the method includes: recommending a route for the vehicle to take that optimizes travel time and distance for navigation using the navigation router and navigates the vehicle in case of an autonomous vehicle or communicates the route to a vehicle operator.” (Para 0016)
“According to several aspects, a method to interpret road signs comprises: collecting image data from a front-facing camera mounted on or in a vehicle including image data of multiple road signs; feeding the image data from the front-facing camera into a first convolutional neural network (CNN) to yield a set of sign predictions including one or more sign instances, and in parallel feeding the image data from the front-facing camera into a second CNN defining a text extractor to extract text candidates including multiple text instances; providing sign and sign data localization for the second CNN to compute a text order from the multiple text instances; transferring individual sign instances from the first CNN and individual ones of the text instances in digitized forms from an optical character recognizer (OCR) into a sign text synthesizer module; and forwarding an output of a semantic encoding and interpretation module to a planner; and operating the planner to compute a route plan and feeding the route plan to a navigation router.” (Para 0021)
“In another aspect of the present disclosure, the method includes: recommending a route for the vehicle to take that optimizes travel time and distance for navigation using the navigation router; and communicating the route to a map which identifies and presents the route with updated path constraints to a vehicle operator.” (Para 0023)
Wang does not explicitly teach:
one or more text prompts [[an]] input [[in]]to one or more language
Bhattacharyya, in the same field of endeavor of prompt generation, teaches:
one or more text prompts [[an]] input [[in]]to one or more language
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Claim 10:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu further teaches:
wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system for generating synthetic data using AI; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.
(Xu) – “Referring to FIG. 1, a road sign interpretation system 10 includes a camera 12 provided in or on a vehicle 14 such as an autonomous vehicle, a battery electric vehicle, a gasoline engine automobile vehicle, a truck, a van, a sport utility vehicle, or the like.” (Para 0034)
“A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
Examiner Note: Road Sign Interpretation System corresponds with a perception system for an autonomous or semi-autonomous machine. A convolutional neural network is a form of deep learning.
Claim 11:
Xu Explicitly teaches:
A system comprising one or more processors to:
(Xu) – “The road sign interpretation system 10 utilizes an on-board computer 26 which is programmed to operate and perform a method of operating the road sign interpretation system 10. The on-board computer 26 receives the image data 16 noted above from the camera 12, as well as other data such as data saved and retrieved from a memory 28 to perform road sign interpretation and to generate and display data results. The on-board computer 26 described in reference to FIG. 1 is programmed to perform operations related to a system and method to collect image data and interpret road signs for the road sign interpretation system 10. The on-board computer 26 is a non-generalized, electronic control device having a preprogrammed digital controller or processor, the memory 28 or a similar non-transitory computer readable medium used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver or input/output ports. The computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.” (Para 0035)
receive image data representing one or more objects in an environment;
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
extract one or more first natural language characters that were represented in the image data;
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
provide a representation of the one or more first natural language characters extracted from the image data as [at least a portion of one or more text prompts [[an]] input [[in]]to one or more language
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
Examiner Note: Bracketed text not explicitly taught by primary reference, but is taught by non-primary reference later in the rejection.
cause presentation, using a device associated with an operator or occupant of an ego-machine, of a representation of the [one or more second natural language characters that were generated by the one or more language models].
(Xu) – “In another aspect of the present disclosure, an on-board computer receives the image data from the camera, in addition to data saved and retrieved from a memory of the on-board computer to perform road sign interpretation and to generate and display data results.” (Para 0007)
“The camera 12 may be any type of color camera and is forward or front-facing to receive a data set having image data 16 including data defining a roadway 18 upon which the vehicle 14 is traveling on and toward, data of one or more other vehicles 20 also on the roadway 18, and data of at least one road sign 22 providing information to an occupant or operator of the vehicle 14 related to the roadway 18. According to several aspects, the data set of the image data 16 received by the camera 12 may be live displayed on a screen 24 of the camera 12 or displayed in a similar display provided in the vehicle 14.” (Para 0034)
Examiner Note: Bracketed text not explicitly taught by primary reference, but is taught by non-primary reference later in the rejection.
Xu does not explicitly teach:
at least a portion of one or more text prompts [[an]] input [[in]]to one or more language that were generated by the one or more language models
Wang, in the same field of endeavor of image analysis, teaches:
one or more second natural language characters… one or more second natural language characters
(Wang) – “Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
PNG
media_image1.png
350
368
media_image1.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the apparatus for driving assistance of Wang. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “the AR technology can be applied in the vehicle travelling process to assist a driver in better mastering driving information in the vehicle travelling process, and the user experience can be improved.” (Wang Abstract)
Wang does not explicitly teach:
at least a portion of one or more text prompts [[an]] input [[in]]to one or more language that were generated by the one or more language models
Bhattacharyya, in the same field of endeavor of prompt generation, teaches:
at least a portion of one or more text prompts [[an]] input [[in]]to one or more language that were generated by the one or more language models
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Claim 12:
Rejected using the same rationale as Claim 2
Claim 15:
Rejected using the same rationale as Claim 10
Claim 16:
Xu Explicitly teaches:
A method comprising: receiving image data representing one or more objects in an environment, the image data generated using one or more first sensors of an ego-machine traversing the environment;
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
extracting one or more first natural language characters represented in the image data;
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
generating, [based at least on applying one or more text prompts comprising the first set of one or more natural language characters that were extracted from the image data to [[via]] a language model a second set of one or more natural language characters] responsive to
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
“Referring to FIG. 2 and again to FIG. 1, the camera 12 of the road sign interpretation system 10 receives the image data 16 from the camera 12. The image data 16 is compared to common road sign text saved in and retrieved from the memory 28 of the on-board computer 26, including for example road sign text such as “one-way” text 30 from a road sign 32. A fixed size feature vector of the common road sign text saved in the memory 28 is compared to a fixed size feature vector of text collected by the camera 12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system 10 is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image 34 presented as a digital image on an electronic, digital sign 36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.” (Para 0036)
Examiner Note: Bracketed text not explicitly taught by primary reference, but is taught by non-primary reference later in the rejection.
causing presentation, at a device associated with an operator of the ego-machine, of a representation of the [one or more second natural language characters that were generated by the language model].
(Xu) – “In another aspect of the present disclosure, an on-board computer receives the image data from the camera, in addition to data saved and retrieved from a memory of the on-board computer to perform road sign interpretation and to generate and display data results.” (Para 0007)
“The camera 12 may be any type of color camera and is forward or front-facing to receive a data set having image data 16 including data defining a roadway 18 upon which the vehicle 14 is traveling on and toward, data of one or more other vehicles 20 also on the roadway 18, and data of at least one road sign 22 providing information to an occupant or operator of the vehicle 14 related to the roadway 18. According to several aspects, the data set of the image data 16 received by the camera 12 may be live displayed on a screen 24 of the camera 12 or displayed in a similar display provided in the vehicle 14.” (Para 0034)
Examiner Note: Bracketed text not explicitly taught by primary reference, but is taught by non-primary reference later in the rejection.
Xu does not explicitly teach:
based at least on applying one or more text prompts comprising the first set of one or more natural language characters extracted from the image data to [[via]] a language model a second set of one or more natural language characters… the second set of one or more natural language characters that were generated by the language model
Wang, in the same field of endeavor of image analysis, teaches:
a second set of one or more natural language characters… the second set of one or more natural language characters
(Wang) – “Further, the step of displaying virtual 3D display information corresponding to the acquired traffic signs within the road area that the current vehicle has passed by comprises: adjusting, according to the current position of the vehicle and the virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, virtual 3D display information corresponding to the traffic signs within the road area that the current vehicle has passed by, and displaying the adjusted virtual 3D display information corresponding to the traffic signs. Further, the step 102 comprises: determining a display mode corresponding to virtual 3D display information; and displaying, based on the determined display mode, virtual 3D display information corresponding to the driving assistance information. The display mode comprises at least one of the following: the display position of virtual 3D display information, the display pose of virtual 3D display information, the display size of virtual 3D display information, the display starting time of virtual 3D display information, the display ending time of virtual 3D display information, the display duration of virtual 3D display information, the display detailedness of contents of virtual 3D display information, the presentation mode for virtual 3D display information, and the display relationship between multiple pieces of virtual 3D display information. The presentation mode comprises at least one of the following: words, icons, animations, sound, light and vibration.” (Col 8 Ln 47- Col 9 Ln 8)
PNG
media_image1.png
350
368
media_image1.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the apparatus for driving assistance of Wang. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “the AR technology can be applied in the vehicle travelling process to assist a driver in better mastering driving information in the vehicle travelling process, and the user experience can be improved.” (Wang Abstract)
Wang does not explicitly teach:
based at least on applying one or more text prompts comprising the first set of one or more natural language characters extracted from the image data to [[via]] a language model… that were generated by the language model
Bhattacharyya, in the same field of endeavor of text prompt generation, teaches:
based at least on applying one or more text prompts comprising the first set of one or more natural language characters extracted from the image data to [[via]] a language model … that were generated by the language model
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Claim 17:
Rejected using the same rationale as Claim 2
Claim 20:
Rejected using the same rationale as Claim 10
Claim(s) 3, 13, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu (US20240062555) in view of Wang (US11767024) further in view of Bhattacharyya (US20250119625) further in view of Scott (US20240428684).
Claim 3:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu does not explicitly teach the following limitations in full. Bhattacharyya further teaches:
wherein the processing circuitry is
and generate the one or more text prompts comprising the representation of the one or more first natural language characters [based at least on [[the]] determining that [[the]]a speed of the ego-machine is below or exceeds [[the]] a threshold speed.]
(Bhattacharyya) – “Other example of text data that may be identified or generated include video descriptions, video captions, video objects, and/or other types of data that may be determined based on analysis of the video, or portion thereof. Accordingly, in embodiments, the text data obtainer 220 may facilitate, reference, or use technology that extracts information from a video (e.g., via video frames). For example, types of information that may be extracted from video frames includes literal text present on the frame and scene understanding of the frames.” (Para 0061)
“To extract textual information present in a video frame, also referred to herein as a video description, one technology that may be used is OCR, such as PP-OCR. Text information present in the frames can reinforce the message present in a scene and inform viewers on what to expect next. In some cases, the OCR text can be filtered and only unique words are used for further processing.” (Para 0062)
“Turning initially to method 600 of FIG. 6, method 600 is directed to facilitating efficient generation of video insights, in accordance with embodiments of the present technology. Initially, at block 602, text data associated with a video is obtained. As described herein, the text data can correspond with a plurality of modalities of the video. For example, some text data corresponds with the audio of the video, while other text data corresponds with images of the video. Examples of text data includes video metadata, video descriptions, video captions, video objects, and a video transcription. The video descriptions, video captions, and/or video objects can be identified in association with keyframes extracted from the video. In some cases, keyframes to analyze may be identified using an optical flow-based approach and/or a sampling-based approach.” (Para 0124)
“At block 604, a model prompt to be input into a large language model is generated. In embodiments, the model prompt includes the text data associated with the video. The model prompt can be generated by concatenating different types of text data, such as, for example, metadata, descriptions, captions, objects, transcriptions, etc. At block 606, a text representation that represents the video in natural language based on the text data is obtained. Generally, the text representation is generated using a large language model. The text representation may be of any length, but generally is intended to be a shortened representation of the video (as compared to the video transcript) such that relevant aspects are included in the text representation while irrelevant aspects are no included in the text representation. At block 608, the text representation is provided as input into a machine learning model to generate a video insight that indicates context of the video.” (Para 0125)
PNG
media_image2.png
624
354
media_image2.png
Greyscale
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the method for facilitating efficient generation of video insights of Bhattacharyya. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “improve computing resource consumption, such as computer memory and latency at least because not as much data (e.g., parameters) is stored or used for producing the model output and computational requirements otherwise needed for fine-tuning are not needed.” (Bhattacharyya Para 0022)
Willmott does not explicitly teach:
based at least on [[the]] determining that [[the]]a speed of the ego-machine is below or exceeds [[the]] a threshold speed.
Scott, in the same field of endeavor of vehicle control, teaches:
based at least on [[the]] determining that [[the]]a speed of the ego-machine is below or exceeds [[the]] a threshold speed.
(Scott) – “wherein the image of the parking area is received in response to the vehicle traveling at a speed below a predetermined threshold.” (Claim 4)
“As shown in FIG. 5 process 500 may include process storing a metadata associated with the image of the parking area (block 502). For example, process 500 may receive metadata associated with the one or more image sensors wherein the metadata associated with the image of the parking area includes at least one of: a time information, a location information, a lighting condition, a vehicle speed, a vehicle user, or any appropriate metadata associated with the images received from the one or more image sensors.” (Para 0043)
“Method may include: storing a metadata associated with the image of the parking area; training a machine learning model based on a stored image of the parking area, a stored number associated with the available parking spot, and the stored metadata associated with the image of the parking area; receiving, from the vehicle, a vehicle location and a current time; determining, by the machine learning model, at least one available parking spot based on the vehicle location and the current time; and displaying to an user, via the vehicle display, at least one available parking spot in the parking area associated with the vehicle location.” Para 0006)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the parking spot detection of Scott. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “The user would be saved the effort of un-parking the vehicle to determine the identification number and re-parking it before returning to the payment kiosk.” (Scott Para 0015)
Claim 13:
Rejected using the same rationale as Claim 3
Claim 18:
Rejected using the same rationale as Claim 3
Claim(s) 4, 14, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu (US20240062555) in view of Wang (US11767024) further in view of Bhattacharyya (US20250119625) further in view of Hann Wang (US20250006196).
Claim 4:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu does not explicitly teach the following limitations. Hann Wang, in the same field of endeavor machine learning teaches:
one or more text prompts include[[s]] a query and one or more of: a 1-shot example of one or more representative outputs; one or more few-shot examples of one or more representative outputs, entity data associated with the one or more first natural language characters, or a hierarchical data structure representing multiple features of the environment.
(Hann Wang) – “In embodiments where the language model 160 is an LLM, the input to the LLM may be in the form of a prompt. A prompt may be a natural language input, for example, an instruction, for the LLM to generate an output according to the prompt. The output generated by the LLM may be a natural language output responsive to the prompt.” (Para 0023)
“The language model 160 may be configured using various learning techniques. For example, in some embodiments, the language model 160 may be configured using few-shot learning. In few-shot learning, the model learns how to learn to solve the given problem. In this approach, the model is provided with a limited number of examples (i.e., “few shots”) from the new task, and the model uses this information to adapt and perform well on that task. Few-shot learning may require fewer amount of training data than implementing other fine-tuning techniques. For further example, in some embodiments, the language model 160 may be configured using one-shot learning, which is similar to few-shot learning, except the model is provided with a single example.” (Para 0024)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the Techniques for generating a prompt for a language model of Hann Wang. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “Providing such a prompt further allows the language model to accurately determine an appropriate action to be performed that is responsive to the user input.” (Hann Wang Para 0018)
Claim 14:
Rejected using the same rationale as Claim 4
Claim 19:
Rejected using the same rationale as Claim 4
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu (US20240062555) in view of Wang (US11767024) further in view of Bhattacharyya (US20250119625) further in view of Agrawal (US20230061784).
Claim 7:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu further teaches:
wherein the one or more objects include a traffic sign, and wherein the one or more processing circuitry is further to:
(Xu) – “A road sign interpretation system includes a front-facing camera mounted on or in a vehicle collecting image data of multiple road signs. A first convolutional neural network (CNN) receives the image data from the front-facing camera and yields a set of sign predictions including one or more sign text instances. A second CNN defining a text extractor receives the image data from the front-facing camera and extracts text candidates including the multiple sign text instances. Sign and sign data localization is provided in the second CNN to compute a text order from the multiple sign text instances. A sign text synthesizer module receives individual sign text instances from the first CNN and individual ones of the sign text instances in digitized forms from an optical character recognizer (OCR). A semantic encoding and interpretation module receives the sign text instances and identifies semantics of the multiple road signs.” (Abstract)
Xu does not explicitly teach:
based at least on the first image data and second image data representing one or more portions of the operator one or more text prompts input [[in]]to the one or more language language
Agrawal, in the same field of endeavor of image analysis, teaches:
based at least on the first image data and second image data representing one or more portions of the operator
(Agrawal) – “The right and left camera apertures 126 and 128 may capture visual data relevant to the outward scene of a vehicle (e.g., through side windows of the vehicle, images appearing in side-view mirrors, etc.) and/or may capture visual data relevant to the inward scene of a vehicle (e.g., a part of the driver/operator, other objects or passengers inside the cab of a vehicle, objects or passengers with which the driver/operator interacts, etc.).” (Para 0036)
“The driving scenario detailed in FIG. 3 illustrates the compounding risks that may occur when a distracted driving event occurs at the time that overlaps with or occurs within a short time of a traffic violation. In this example, the traffic violation was a failure to come to a complete or partial stop at a valid stop sign, which may be referred to as a no-stop stop sign violation. While a failure to observe stop signs may be understood to correlate with collision risk on its own, it may also be appreciated that a collision at a stop sign could still be avoided if the driver is attentive to the movements of other vehicles in the intersection. In the example illustrated in FIG. 3, both vehicles were moving slowly enough that either driver should have been able to come to a complete stop and avoid the collision had he or she been aware of the movements of the other vehicle. When the driver's attention is focused elsewhere, however, such that she fails to notice the movements of another vehicle, the risk of collision is substantially higher. The compounding risks of these two events may be considered super-linear in the sense that the risk associated with jointly observed no-stop stop sign violation and distracted driving may be greater than the risk of a no-stop sign violation observed at one time added to the risk associated with distracted driving at another time.” (Para 0061)
“Similarly, according to certain aspects, a determination that a driver is distracted may modify thresholds at which other alerts are triggered. Driver distraction may be based on a determination that the driver is eating, is talking, is singing, and the like. In some embodiments, driver distraction may be determined based on a paucity of saccadic eye movements, indicating that the driver is not actively scanning a visual scene. In these examples, the driver may be looking in the direction of the vehicle ahead but may still be considered distracted. The threshold at which a following distance alert is triggered may be set to a more sensitive level than it would be if the driver were not determined to be distracted.” (Para 0098)
“In some embodiments, a quality of an in-cab alert may be modified. For example, if the driver receives an in-cab alert at a time corresponding to a modified threshold, the tone of the alert (if audible) may be different than it would be if no threshold modifiers applied. In this way, the potentially distracted driver may be given a notification that he or she should attend more closely to the driving scene.” (Para 0099)
Examiner Note: Per BRI, risk corresponds with score.
provide a representation of the score as at least a second portion of the one or more text prompts input [[in]]to the one or more language language
(Agrawal) – “Certain aspects of the present disclosure may be directed to training a machine learning model, such as a neural network model, based at least in part on examples of combination alerts. In one example, combination alerts for which a detected Hard Braking event is preceded by Driver Distraction may be used to train a model to learn to predict when control of the vehicle should be taken from the driver.” (Para 0100)
“Intelligent in-cab warnings may help prevent or reduce vehicular accidents. In-cab warnings of unsafe events before or during the traffic event may enable the driver to take action to avoid an accident. In-cab messages that are delivered shortly after unsafe events have occurred may still be useful for the driver in that, in comparison to a delay of several hours or days, a message presented soon after an event is detected by an in-vehicle safety device may enhance the learning efficacy of the message. For example, the driver may to self-coach and learn from the event and how to avoid similar events in the future.” (Para 0038)
“Similarly, according to certain aspects, a determination that a driver is distracted may modify thresholds at which other alerts are triggered. Driver distraction may be based on a determination that the driver is eating, is talking, is singing, and the like. In some embodiments, driver distraction may be determined based on a paucity of saccadic eye movements, indicating that the driver is not actively scanning a visual scene. In these examples, the driver may be looking in the direction of the vehicle ahead but may still be considered distracted. The threshold at which a following distance alert is triggered may be set to a more sensitive level than it would be if the driver were not determined to be distracted.” (Para 0098)
“In some embodiments, a quality of an in-cab alert may be modified. For example, if the driver receives an in-cab alert at a time corresponding to a modified threshold, the tone of the alert (if audible) may be different than it would be if no threshold modifiers applied. In this way, the potentially distracted driver may be given a notification that he or she should attend more closely to the driving scene.” (Para 0099)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the intelligent driving monitoring systems of Agrawal. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “Combinations of co-occurring driving events may be detected and used to warn on anomalies, prevent accidents, provide feedback to the driver, and in general provide a safer driver experience.” (Agrawal Abstract)
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu (US20240062555) in view of Wang (US11767024) further in view of Bhattacharyya (US20250119625) further in view of Ingebretsen (US20230344537).
Claim 8:
Xu in combination with the references relied upon in Claim 1 teach those respective limitations. Xu does not explicitly teach the following limitations. However, Ingebretsen, in the same field of endeavor of language processing teaches:
wherein the processing circuitry is
(Ingebretsen) – “Upon the detection of a current location being relatively close to a major city or metropolitan area, or an area having a larger amount of detected radio stations (e.g., above a threshold level), the broadcast information may be received and the vocabulary updated as a result. The broadcast information may also be received in response to a determination that the user's location has become relatively stationary for a predetermined period of time. For example, the user may be traveling but may stop at a rest location for a threshold period of time (e.g., at least 30 minutes). As a result, the broadcast information may be received and the vocabulary updated accordingly.” (Para 0244)
“A speech input 806 received at device 802 includes the command “Tune to 101.5 FM.” Based on the modified vocabulary, a user intent of speech input 806 is determined. In particular, given the known radio station address of “101.5 FM,” speech input 806 may be correctly interpreted as a command to navigate to the specific radio station indicated by the user using the radio device installed within the vehicle. Accordingly, a command can be sent to the computing device within the vehicle to navigate to station “101.5 FM.” Touch-screen interface 804 may then be updated to reflect navigation to the respective station along with the radio station name “The Mix” and currently playing media “Band 1 — Song 1.” The displayed information may be obtained at least in part based on the metadata within the received radio broadcast information (e.g., originating from the computing device within the vehicle and/or the media server). In some embodiments, the display of device 802 is also updated to reflect the radio station information and currently playing media. Device 802 may also display additional information not displayed on touch-screen interface 804, such as an album name corresponding to the respective playing media.” (Para 0242)
provide a representation of the audio data as at least a second portion of the one or more text prompts input [[in]]to the one or more language language
(Ingebretsen) – “Upon the detection of a current location being relatively close to a major city or metropolitan area, or an area having a larger amount of detected radio stations (e.g., above a threshold level), the broadcast information may be received and the vocabulary updated as a result. The broadcast information may also be received in response to a determination that the user's location has become relatively stationary for a predetermined period of time. For example, the user may be traveling but may stop at a rest location for a threshold period of time (e.g., at least 30 minutes). As a result, the broadcast information may be received and the vocabulary updated accordingly.” (Para 0244)
“A speech input 806 received at device 802 includes the command “Tune to 101.5 FM.” Based on the modified vocabulary, a user intent of speech input 806 is determined. In particular, given the known radio station address of “101.5 FM,” speech input 806 may be correctly interpreted as a command to navigate to the specific radio station indicated by the user using the radio device installed within the vehicle. Accordingly, a command can be sent to the computing device within the vehicle to navigate to station “101.5 FM.” Touch-screen interface 804 may then be updated to reflect navigation to the respective station along with the radio station name “The Mix” and currently playing media “Band 1 — Song 1.” The displayed information may be obtained at least in part based on the metadata within the received radio broadcast information (e.g., originating from the computing device within the vehicle and/or the media server). In some embodiments, the display of device 802 is also updated to reflect the radio station information and currently playing media. Device 802 may also display additional information not displayed on touch-screen interface 804, such as an album name corresponding to the respective playing media.” (Para 0242)
“Further, each ASR system 758 includes one or more speech recognition models (e.g., acoustic models and/or language models) and implements one or more speech recognition engines. Examples of speech recognition models include Hidden Markov Models, Gaussian-Mixture Models, Deep Neural Network Models, n-gram language models, and other statistical models.” (Para 0204)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the road sign interpretation system of Xu with the language processing with radio devices of Ingebretsen. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “an improved system for language processing with radio devices is desired.” (Ingebretsen Para 0004)
Response to Arguments
Applicant’s arguments with respect to the 35 U.S.C. 103 rejections mailed 04/16/2026 have been considered but are not convincing.
Specifically, all claims are now rejected in view of Bhattacharyya rather than Willmott, in response to amendment. Examiner maintains that Bhattacharyya resolves any alleged deficiencies within the prior art of record as evidenced in the above rejection rationale.
Therefore, all outstanding claims remain rejected over the prior art.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID RUBEN PEDERSEN whose telephone number is (571)272-9696. The examiner can normally be reached M-Th: 07:00 -16:00 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID RUBEN PEDERSEN/Examiner, Art Unit 3658
Prosecution Insights