DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Drawings
The drawings are objected to because
Structures of the claimed system are not shown in Figures properly. Structures have to be shown in figures showing proper labels.
Figs.1, 2, 4, 5 shows some boxes without proper labels. Reference number as well as label of the structure or steps should be included in these figures for proper understanding of the figures.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Claims 1-10 are interpreted under 35 USC 112(f) because they recite a generic place holder “a vehicle control module” that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Applicant provides the hardware support of the generic place holder in specification [0087] “In this application, including the definitions below, the term "module" or the term "controller" may be replaced with the term "circuit." The term "module" may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described function”.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 9, 11 and 18 are rejected under 35 U.S.C. 103) as being unpatentable over Kale et al. ( US patent publication 20210319219, “Kale”) in view of Ohashi et al. ( US patent publication: US 20230298211, “Ohashi”).
Regarding claim 1, Kale teaches, as system (Fig. 1) for color adjustment of an augmented reality vehicle display, the system comprising
a windshield of a vehicle; (“[0011]…..For example, an augmented reality (AU) display of the traffic signals/signs can be presented on the heads up display, the windshield,”)
a display interface configured to display images on the windshield; ([0019]…. The host system (104) then transmits this visualization to one or more of the display modules (149a, 149b, 149c, 149d) for display.”)
a front vehicle camera ( element 109) configured to obtain an image; (“[0018]…. The image sensor (109) captures an image of the environment, including the signal (180),”) and
a vehicle control module (Integrated image sensing device 101 and Host system 104) configured to:
access the image from the front vehicle camera; (“[0087] In the illustrated embodiment, capturing an image comprises capturing an image (or frame of video) using an image sensor installed on a vehicle. In one embodiment, the method captures the image via an image sensor installed in a vehicle.”)
process the image to determine an object of interest in the image; (“[0088] In some embodiments, the image captures an area of interest. Additionally, in some embodiments, the area of interest may or may not include an object of interest. As used herein, an object of interest generally refers to any object appearing in the area of interest. In some embodiments, the object of interest comprises a pre-defined object set in the memory of the vehicle. For example, the object of interest may comprise an object from a set of objects corresponding to traffic signals, signs, markers, or other shapes appearing on a roadway. In general, an object of interest can be defined by defining an AI/ML model that identifies an object of interest. Thus, in some embodiments, the method can be configured to detect any object of interest that can be modeled using an AI/ML model.”)
supply the image to a trained machine learning model to generate a selection output according to at least one background color or object color
identified in image, which contrasts the at least one background color or object color with respect to viewing by a driver; (Step 901 trains a machine learning model. Step 903 and Step 905 determines generates an output or and in step 903, 905 and 907 determines alternative representation. “[0100] FIG. 9 is a flow diagram illustrating a method for training and using a machine learning model for displaying an alternative representation of a detected object according to one embodiment. [0101] In the previous method described in FIG. 8, alternative representations are displayed for all detected objects (or a subset thereof). However, for some drivers, less than all objects may need alternative representations. The following method provides alternatives to reduce the number of alternative representations display to the driver.……. …….[0105] In block 903, the method processes a current image using the AI engine. In the illustrated embodiment, the processing of block 903 comprises inputting the current image into the model trained in block 901. The output of the model comprises a confidence level indicating whether the user will not detect the object or not based on the user's level of color blindness. [0106] In block 905, the method determines if an alternative representation is needed. [0107] In one embodiment, the method analyzes the confidence level of the output of the AI model using the model trained in block 901. If the confidence level is low, indicating that the user will not detect an object in the given image, the method will proceed to block 907. Otherwise, the method will forego generating alternative representations. [0108] In block 907, the method displays the alternative representation in one or more display modules.”) but doesn’t teach generate a color selection output wherein the color selection output indicates a specified color, wherein the color selection output indicates a specified color.
However, Ohashi teaches, supply the image to a trained machine learning model to generate a selection output according to at least one background color or object color identified in image, wherein the color selection output indicates a specified color which contrasts the at least one background color or object color, ([0018] “Some embodiments of this disclosure may use an artificial intelligence (AI) image recognition model (AI model) to simulate how a color vision deficient person recognizes a particular object in a particular image. Some embodiments may use the output of the AI model to assist color deficient users in recognizing otherwise-unrecognizable object(s) by changing its/their color(s), thus making it possible to create images with reduced color barriers”, Fig. 6 element 600 provides steps for the detecting a changed color for an object that has different contrast than the object image. Fig. 6 steps 635 determines an alternate color that can be recognized by color deficient person which has different contrast than the object image.)
Kale and Ohashi are analogous as they are from the field of image correction for display.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Kale to have the trained machine learning model to generate of a color selection output according to at least one background color or object color identified in image wherein the color selection output indicates a specified color which contrasts the at least one background color or object color as taught by Ohashi.
The motivation to include the modification is to provide alternative color of an object that can be better recognized by driver.
Kale as modified by Ohashi teaches, display an object identifier on the windshield according to a gaze direction of the driver, via the display interface, wherein the object identifier includes the color selection output from the trained machine learning model.( Kale displays alternate representation of the object to windshield display which isa display on windshield according to a gaze direction. Kale “[0107]…..the method will proceed to block 907. Otherwise, the method will forego generating alternative representations. [0108] In block 907, the method displays the alternative representation in one or more display modules. This alternative representation is modified with Ohashi to get an object identifier with changed color and the object identifier includes the color selection output from the trained machine learning model.)
Claim 11 is directed to a method and its steps are similar in scope and function of the elements of the device claim 1 and therefore claim 11 is rejected with same rationales as specified in the rejection of claim 1.
Regarding claims 9 and 18, Kale as modified by Ohashi teaches, wherein displaying the object identifier includes displaying the object identifier on the windshield during a first time period, (Kale, “ [0098] The following example of a traffic signal (light) is provided to illustrate the operation of the method of FIG. 8. As a user approaches the light, the camera records images (801) of the light (which is increasing in size as the user approaches). The camera identifies the object using the AI engine and determines its state based on the colors in the image (803). Initially, the light may be “green” and the method will load an alternative representation of a “green traffic light” which, as one example, may comprise an audible signal (“green light in X meters”, where X is the distance to the light and can be identified based on the size of the traffic light and the user's speed or based on a roadway database) (805, 807).” Ohashi’s alternative representation based on color section output is integrated already. ) and
the vehicle control module is configured to: subsequent to the first time period, access a second image from the front vehicle camera;supply the second image to a trained machine learning model to generate a second color selection output according to at least one background color or object color identified in the second image; and update color of the object identifier on the windshield, via the display interface, in response to the second color selection output being different than a previous color selection output. (“[0098]……As the driver continues to approach the light, the light will change to “yellow.” The method repeats the above steps and may issue an audible alert (“yellow light in X meters”). Additionally, the method may display a color-neutral warning on the dash (“Yellow Light Ahead”). Ohashi’s alternative representation based on color section output is integrated already)
Claim(s) 2 and 12 are rejected under 35 U.S.C. 103) as being unpatentable over Kale as modified by Ohashi and further in view of Radu et al. ( US patent: 11971548, “Radu”).
Regarding claims 2 and 12, Kale as modified by Ohashi doesn’t expressly teach, wherein :the display interface includes a projector; and displaying the object identifier includes projecting the object identifier on the
windshield as an augmented reality heads-up display.
However, Radu teaches, the display interface includes a projector; and displaying the object identifier includes projecting an object identifier on the
windshield as an augmented reality heads-up display. (Column 4 Lines 15-30: The screen 150 may be an organic light-emitting diode (OLED) screen fitted over the windscreen 112, or attached to an inner face of windscreen 112. Rather than an OLED screen overlaid on the windscreen 112, the screen 150 may have an associated projector to project the marker 142 onto the screen 150, wherein the screen 150 reflects visible light from the projector 152. See FIG. 2. Alternatively, the screen 150 and the windscreen 112 may be a single unitary structure, rather than two independent structures layered together. A windscreen 112 may be shatter-proof automobile glass. The screen 150 may be a see-through screen that produces images from within (e.g., OLED), or reflect images projected onto it (e.g., projector).” Alternatively, the marker 142 may be displayed on a screen 150/windscreen 112 with light provided by a projector 152.)
Kale as modified by Ohashi and Radu are analogous as they are from the field of display device of vehicle.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Kale as modified by Ohashi to have the display interface includes a projector; and displaying the object identifier includes projecting an object identifier on the windshield as an augmented reality heads-up display as taught by Radu.
The motivation to include the modification is to use an alternative display in a vehicle..
Claim(s) 3 and 13 are rejected under 35 U.S.C. 103) as being unpatentable over Kale as modified by Ohashi and further in view of Elchhorn et al. ( EP 3983251 B1, “Elchhorn”).
Regarding claims 3 and 13, Kale as modified by Ohashi doesn’t expressly teach :the display interface includes multiple micro light emitting diodes embedded in or adjacent the windshield; and displaying the object identifier includes displaying the object identifier on the windshield via the multiple micro light emitting diodes
Elchhorn teaches, the display interface includes multiple micro light emitting diodes embedded in or adjacent the windshield; (“Display system for a motor vehicle having a display device (6) with multiple display elements (61) arranged on a top side (31) of a dashboard (3) so that a display image is reflected from a reflection region (21) into an eye region of a driver and/or another vehicle occupant, characterized in that the multiple display elements (61) are in the form of micro-LED display devices set up by field arrangement of individually actuatable micro-LED elements, wherein the display elements (61) are arranged adjacently to one another over at least a subregion of the width of the dashboard (3), wherein the display elements (61) adjoin a lower edge of the windscreen (2), wherein the display elements (61) have a rectangular surface area, which means that each of the display elements (61) has at least one corner adjoining the windscreen “)
Kale as modified by Ohashi and Elchhorn are analogous as they are from the field of display device of vehicle.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Kale as modified by Ohashi to have the display interface include multiple micro light emitting diodes embedded in or adjacent the windshield as taught by Elchhorn.
The motivation to include the modification is to use an alternative display that has advantage of selectively turning on or off the lighting elements
. Kale as modified by Ohashi and Elchhorn teaches, displaying the object identifier includes displaying the object identifier on the windshield via the multiple micro light emitting diodes. (Kale displays alternate representation of the object to windshield display which isa display on windshield according to a gaze direction. Kale [0107]…..the method will proceed to block 907. Otherwise, the method will forego generating alternative representations. [0108] In block 907, the method displays the alternative representation in one or more display modules. This alternative representation is modified with Ohashi to get an object identifier with changed color and the object identifier includes the color selection output from the trained machine learning model.”)
Claim(s) 4-5 and 14 are rejected under 35 U.S.C. 103) as being unpatentable over Kale as modified by Ohashi and further in view of IM et al. ( US patent publication: 20210331681, “IM”).
Regarding claims 4 and 14, Kale as modified by Ohashi doesn’t expressly teach, an orientation sensor configured to detect a gaze direction of at least one of eyes of the driver of the vehicle or a head of the driver of the vehicle, wherein the vehicle control module is configured to determine the gaze direction of the driver via the orientation sensor.
However IM teaches, an orientation sensor configured to detect a gaze direction of at least one of eyes of the driver of the vehicle or a head of the driver of the vehicle, wherein the vehicle control module is configured to determine the gaze direction of the driver via the orientation sensor. (“[0025] An intellectual computing device for controlling a vehicle according to another aspect of the present invention includes: a camera disposed in the vehicle; a head-up display; a sensing unit including at least one sensor; a processor; and a memory including a command that can be executed by the processor, in which the command: outputs a primary warning by showing a virtual object on a windshield of the vehicle through the head-up display when recognizing a drowsy state of a driver on the basis of state information of the driver acquired through the sensing unit; and acquires a gaze response speed of the driver to the virtual object by tracing the gaze of the driver through the camera, provides feedback according to the gaze response speed, and outputs a secondary warning and controls the vehicle in accordance with the secondary warning when determining that the gaze response speed is lower than a predetermined reference.”)
Kale as modified by Ohashi and IM are analogous as they are from the field of display device of vehicle.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Kale as modified by Ohashi to have included an orientation sensor configured to detect a gaze direction of at least one of eyes of the driver of the vehicle or a head of the driver of the vehicle, wherein the vehicle control module is configured to determine the gaze direction of the driver via the orientation sensor as taught by IM.
The motivation to include the modification is to provide display at proper location based on gaze of the user for easy readability by the driver.
Regarding claim 5, Kale as modified by Ohashi and IM wherein the orientation sensor includes a gaze tracker camera configured to track at least one of eye movements of the driver and head movements of the driver. (IM, “[0208] The at least one sensor may be at least one camera disposed in the vehicle 10. For example, the camera may be disposed to photograph a driver in front-rear and left-right directions. The processor 170 can use at least one of the number of times of closing eyelids, the open size of eyelids, or a movement speed of eyelids of a driver to determine the state information of the driver by analyzing images acquired from the camera.”)
Claim(s) 7 and 16 are rejected under 35 U.S.C. 103) as being unpatentable over Kale as modified by Ohashi and further in view of et al. ( US patent : 11745754, “Maruyama”).
Regarding claims 7 and 16, Kale as modified by Ohashi doesn’t expressly teach, wherein the object identifier displayed on the windshield includes a bounding box surrounding at least a portion of the object of interest in a field of view of the driver.
Maruyama teaches, the object identifier displayed on the windshield includes a bounding box surrounding at least a portion of the object of interest in a field of view of the driver. (Col 20 Line 38-46: “FIG. 16 illustrates an example of the object radar display to be displayed on the HUD 156 by the attention calling unit 514. The object radar display 520 includes a vehicle icon 522 indicating the vehicle 100, and a ring-shaped display 524 omnidirectionally indicating an entire circumferential area of the vehicle 100 as the center at 360°. The attention calling unit 51 outputs the visual display corresponding to the object direction to the arc part of the ring-shaped display 524.”)
Kale as modified by Ohashi and Maruyama are analogous as they are from the field of vehicle display.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention Kale as modified by Ohashi to have the object identifier displayed on the windshield to include a bounding box surrounding at least a portion of the object of interest in a field of view of the driver as taught by Maruyama.
The motivation to include the modification is to limit the focus area on the HUD or windshield display.
Claim(s) 8 and 17 are rejected under 35 U.S.C. 103) as being unpatentable over Kale as modified by Ohashi and Maruyama and further in view Higuchi et al. ( US patent publication: 20180373027, “Higuchi”).
Regarding claims 8 and 17, Kale as modified by Ohashi and Maruyama teaches, while leaving the object of interest visible in a center of the bounding box. (Maruyama, Col 20 Line 38-46: “FIG. 16 illustrates an example of the object radar display to be displayed on the HUD 156 by the attention calling unit 514. The object radar display 520 includes a vehicle icon 522 indicating the vehicle 100, and a ring-shaped display 524 omnidirectionally indicating an entire circumferential area of the vehicle 100 as the center at 360°. The attention calling unit 51 outputs the visual display corresponding to the object direction to the arc part of the ring-shaped display 524.”) but doesn’t expressly teach, wherein the vehicle control module is configured to fill in a portion of the bounding box with the color selection output, while leaving the object of interest visible in a center of the bounding box.
However, Highuchi teaches, the vehicle control module is configured to fill in a portion of the bounding box with the color selection output,(“[0046] A second feature of the HUD device 200 according to the embodiment is as follows. That is, the HUD device 200 determines whether RGB values of a background color, which is a color in the background of a display image displayed by the HUD device 200, is similar to RGB values of a display color of a display image. As a result, if the both are similar (or match each other), with reference to the correction color LUT 253t, the HUD device 200 corrects the display color on the surroundings of the display image, or on the entire surface of the image display area excluding the display image, to a correction color having RGB values that are neither similar to the RGB values of the background color nor the RGB values of the display color (Steps S6 to S9 in FIG. 8).”)
Higuchi and Kale as modified by Ohashi and Maruyama are analogous as they are from the field of vehicle display.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Kale as modified by Ohashi and Maruyama to have the vehicle control module configured to fill in a portion of the bounding box with the color selection output as taught by Higuchi.
The motivation to include the modification is to provide better readability of object of interest.
Claim(s) 20 is directed under 35 U.S.C. 103) as being unpatentable over Kale in view of Ohashi and Radu.
Regarding claim 20, Kale teaches, A method of adjusting color for an augmented reality vehicle display, the method comprising:
processing an image from a vehicle camera of a vehicle to determine an object of interest in the image; (“[0087] In the illustrated embodiment, capturing an image comprises capturing an image (or frame of video) using an image sensor installed on a vehicle. In one embodiment, the method captures the image via an image sensor installed in a vehicle.” (“[0088] In some embodiments, the image captures an area of interest. Additionally, in some embodiments, the area of interest may or may not include an object of interest. As used herein, an object of interest generally refers to any object appearing in the area of interest. In some embodiments, the object of interest comprises a pre-defined object set in the memory of the vehicle. For example, the object of interest may comprise an object from a set of objects corresponding to traffic signals, signs, markers, or other shapes appearing on a roadway. In general, an object of interest can be defined by defining an AI/ML model that identifies an object of interest. Thus, in some embodiments, the method can be configured to detect any object of interest that can be modeled using an AI/ML model.”)
in response to a color vision deficient driver setting being active, applying daltonization color correction to determine a selection output, wherein the selection output indicates a specified output which contrasts at least one background color or object color identified in the image with respect to viewing by a driver of the vehicle; ( “[0101]…However, for some drivers, less than all objects may need alternative representations. The following method provides alternatives to reduce the number of alternative representations display to the driver. [0102] In block 901, the method trains the AI engine to predict a level of deficiency”.
Step 901 trains a machine learning model. Step 903 and Step 905 determines generates an output or and in step 903, 905 and 907 determines alternative representation for a color vision deficient driver. “[0100] FIG. 9 is a flow diagram illustrating a method for training and using a machine learning model for displaying an alternative representation of a detected object according to one embodiment. [0100] FIG. 9 is a flow diagram illustrating a method for training and using a machine learning model for displaying an alternative representation of a detected object according to one embodiment. [0101] In the previous method described in FIG. 8, alternative representations are displayed for all detected objects (or a subset thereof). However, for some drivers, less than all objects may need alternative representations. The following method provides alternatives to reduce the number of alternative representations display to the driver.……. …….[0105] In block 903, the method processes a current image using the AI engine. In the illustrated embodiment, the processing of block 903 comprises inputting the current image into the model trained in block 901. The output of the model comprises a confidence level indicating whether the user will not detect the object or not based on the user's level of color blindness. [0106] In block 905, the method determines if an alternative representation is needed. [0107] In one embodiment, the method analyzes the confidence level of the output of the AI model using the model trained in block 901. If the confidence level is low, indicating that the user will not detect an object in the given image, the method will proceed to block 907. Otherwise, the method will forego generating alternative representations. [0108] In block 907, the method displays the alternative representation in one or more display modules.”) but doesn’t teach generate a color selection output wherein the color selection output indicates a specified color selection output wherein the color selection output indicates a specified color.
However Ohashi teaches, in response to a color vision deficient driver setting being active, applying daltonization color correction to determine a color selection output, wherein the color selection output indicates a specified color which contrasts at least one background color or object color identified in the image,(“[0018] Some embodiments of this disclosure may use an artificial intelligence (AI) image recognition model (AI model) to simulate how a color vision deficient person recognizes a particular object in a particular image. Some embodiments may use the output of the AI model to assist color deficient users in recognizing otherwise-unrecognizable object(s) by changing its/their color(s), thus making it possible to create images with reduced color barriers”, Fig. 6 element 600 provides steps for the detecting a changed color for an object that has different contrast than the object image. Fig. 6 steps 635 determines an alternate color that can be recognized by color deficient person which has different contrast than the object image.)
Kale and Ohashi are analogous as they are from the field of image correction for display.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Kale to have added in response to a color vision deficient driver setting being active, applying daltonization color correction to determine a color selection output, wherein the color selection output indicates a specified color which contrasts at least one background color or object color identified in the image as taught by Ohashi.
The motivation to include the modification is to provide alternative color of an object that can be better recognized by driver.
Kale as modified by Ohashi teaches, displaying an object identifier on a windshield of the vehicle according to a gaze direction of the driver, wherein the object identifier includes the daltonization color correction. ( Kale displays alternate representation of the object to windshield display which isa display on windshield according to a gaze direction. Kale [0107]…..the method will proceed to block 907. Otherwise, the method will forego generating alternative representations. [0108] In block 907, the method displays the alternative representation in one or more display modules. This alternative representation is modified with Ohashi to get an object identifier with changed color and the object identifier includes identifier includes the daltonization color correction. ) but doesn’t expressly teach so via a display projector of an augmented reality heads-up display of the vehicle.
Radu teaches, displaying an object identifier via a display projector of an augmented reality heads-up display of a vehicle. (Column 4 Lines 15-30: The screen 150 may be an organic light-emitting diode (OLED) screen fitted over the windscreen 112, or attached to an inner face of windscreen 112. Rather than an OLED screen overlaid on the windscreen 112, the screen 150 may have an associated projector to project the marker 142 onto the screen 150, wherein the screen 150 reflects visible light from the projector 152. See FIG. 2. Alternatively, the screen 150 and the windscreen 112 may be a single unitary structure, rather than two independent structures layered together. A windscreen 112 may be shatter-proof automobile glass. The screen 150 may be a see-through screen that produces images from within (e.g., OLED), or reflect images projected onto it (e.g., projector).” Alternatively, the marker 142 may be displayed on a screen 150/windscreen 112 with light provided by a projector 152.)
Kale as modified by Ohashi and Radu are analogous as they are from the field of display device of vehicle.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of the claimed invention to have modified Kale as modified by Ohashi to have via a display projector of an augmented reality heads-up display of the vehicle as taught by Radu.
The motivation to include the modification is to use an alternative display in a vehicle..
Allowable Subject Matter
Claims 6, 10, 15 and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim 6 and 15 are objected because the combination of prior art fails to expressly teach, wherein the vehicle control module is configured to: capture an image of a color check pattern with the front vehicle camera; calculate specified color vectors using the image of the color check pattern; determine a minimized difference between different color vectors; and calculate a color mapping matrix according to the minimized difference between different color vectors
Claims 10 and 19 are objected because the best combination of prior arts fails to expressly teach, wherein: the trained machine learning model is trained according to multiple input vectors; wherein: the trained machine learning model is trained according to multiple input vectors; at least a portion of the multiple input vectors include a defined color contrast ratio between two different colors; and the defined color contrast ratio is calculated according to a relative luminance of a lighter one of the two different colors, a relative luminance of a darker one of the two different colors, and a constant weight value corresponding to a contribution of ambient light to relative luminance values for the two different color vectors.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAPTARSHI MAZUMDER whose telephone number is (571)270-3454. The examiner can normally be reached 8 am-4 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at (571)272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAPTARSHI MAZUMDER/Primary Examiner, Art Unit 2612
Prosecution Insights