DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 7-9. 11-14, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kaita et al. (US 20210001845) and Lim et al. (“Forward collision warning system for motorcyclist using smart phone sensors based on time-to-collision and trajectory prediction”).
Regarding claim 1, Kaita et al. discloses an electronic device comprising:
at least one camera (“The detection unit 17 is a camera that captures an image of what is in front thereof” at paragraph 0027, line 1);
a gyro sensor (“In addition to the function of the forward monitoring unit, the ECU 120 performs processing on a signal detected by a gyro sensor 30 to specify the inclination of the vehicle body” at paragraph 0032, line 1);
memory (“The storage device stores, for example, a program that is executed by the processor” at paragraph 0028, line 8); and
at least one processor operably connected with the at least one camera, the gyro sensor, and the memory (“Each ECU includes, for example, a processor, which is typically a CPU, a storage device such as a semiconductor memory, and an interface with an external device” at paragraph 0028, line 5), and
wherein the processor is configured to:
obtain image information by using the at least one camera (“The camera 17 captures an image of what is in front thereof through the opening or the transparent member” at paragraph 0027, last sentence; “Detection units 16 and 17 that detect what is in front of the vehicle 1 is provided rearward of the front cover 12. In this example, the detection units 16 and 17 are provided at fixed positions relative to the main frames. In the present embodiment, the detection unit 16 is a radar (for example, a millimeter wave radar). However, another type of sensor that can detect forward through the front cover 12, such as a sonar that performs ultrasonic distance measurement, or a lidar that performs image detection and distance measurement using laser light may be employed” at paragraph 0024, line 1);
based on obtaining the image information, obtain, first rotation data on the electronic device by using the gyro sensor (“First, the ECU 120 detects the current inclination of the vehicle body using the gyro sensor 30 (501). The detected inclination includes at least a roll angle and a pitch angle. Next, the ECU 120 detects the target using a sensor, i.e. the radar 16, and detects the coordinates thereof (S503)” at paragraph 0046, line 9);
based on the first rotation data, obtain a rotation value for the image information (“Polar coordinates are transformed to rectangular coordinates as described above in this example. Also, in the case of rectangular coordinates, the origin of the of the coordinates is transformed so as to be translated to the origin O as shown in FIG. 3 and so on as necessary. Next, the ECU 120 corrects the coordinates of the target detected by the radar 16 (S505). Finally, the ECU 120 updates the positional information regarding the target with the corrected coordinates (S507), and terminates processing” at paragraph 0046, fourth to last sentence); and
based on the obtained rotation value, identify at least one object regarding the video information (“The position of the preceding vehicle thus corrected is updated at predetermined time intervals, e.g. time intervals in the range of several tens of milliseconds to 100 milliseconds. The identity of the object is determined based on the closeness and predicted movement of the detected position at each update, or an image captured by the camera 17, the preceding tracking target vehicle is identified, and the position thereof is specified” at paragraph 0054, line 1).
Kaita et al. does not explicitly disclose that the image information is video information, based on obtaining the video information, obtain, second rotation data by using a designated estimation model and based on the first rotation data and the second rotation data, obtain a rotation value for the video information.
Lim et al. teaches an electronic device in the same field of endeavor of motorcycle tilt analysis, comprising:
at least one camera (“Hence the camera sensor will be used as a single sensor used to measure multiple measurements such as relative distance and speed” at section 2, line 7);
a gyro sensor (“To detect the leaning angle, a gyroscope and an accelerometer are required, which can be found in an IMU” at section 4.1, line 4);
memory (memory of processing unit); and
at least one processor operably connected with the at least one camera, the gyro sensor, and the memory (processor of processing unit), and
wherein the processor is configured to:
obtain video information by using the at least one camera (“For the video recording, the video is recorded at 29.97 frames per second” at section 5, line 5);
based on obtaining the video information, obtain, first rotation data on the electronic device by using the gyro sensor (“The first method to detect the leaning angle of the ego vehicle and the assumption is that a lean to the right will correspond to the rightwards trajectory of the ego vehicle, and a lean to the left will correspond to the leftwards trajectory accordingly. To detect the leaning angle, a gyroscope and an accelerometer are required, which can be found in an IMU” at section 4.1, line 1), and second rotation data by using a designated estimation model (“Trajectory prediction is very important in the advanced FCW system because by understanding the trajectory of the detected vehicle, the FCWalgorithm will have more time to alert the ego vehicle if there will be a potential collision. The trajectory prediction input data is based on pixel center coordinates of the detected vehicle using the object detection algorithm in the previous section to obtain time-series data” at section 3, line 1; this determines the vehicle’s heading, which is characterized by direction and steering angle);
based on the first rotation data and the second rotation data, obtain a rotation value for the video information (“Similarly, the roll angle will have a threshold that classify if a bike is leaning to the left, straight or right” at section 5.4, line 8; “The image will be segmented into 3 parts which indicates left, straight or right and these regions will be used to classify where does the trajectory of the vehicle detected fall under” at section 5.4, line 5; when the two directions match, the heading is determined to correspond to the left, right or no rotation).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize image derived data as taught by Lim et al. along with the gyro data of Kaita et al. to ensure that the true heading of the vehicle is reflected in the subsequent processing.
Regarding claim 11, Kaita et al. discloses a method of an electronic device comprising:
obtaining image information by using the at least one camera (“The camera 17 captures an image of what is in front thereof through the opening or the transparent member” at paragraph 0027, last sentence; “Detection units 16 and 17 that detect what is in front of the vehicle 1 is provided rearward of the front cover 12. In this example, the detection units 16 and 17 are provided at fixed positions relative to the main frames. In the present embodiment, the detection unit 16 is a radar (for example, a millimeter wave radar). However, another type of sensor that can detect forward through the front cover 12, such as a sonar that performs ultrasonic distance measurement, or a lidar that performs image detection and distance measurement using laser light may be employed” at paragraph 0024, line 1);
based on obtaining the image information, obtain, first rotation data on the electronic device by using the gyro sensor (“First, the ECU 120 detects the current inclination of the vehicle body using the gyro sensor 30 (501). The detected inclination includes at least a roll angle and a pitch angle. Next, the ECU 120 detects the target using a sensor, i.e. the radar 16, and detects the coordinates thereof (S503)” at paragraph 0046, line 9);
based on the first rotation data, obtain a rotation value for the image information (“Polar coordinates are transformed to rectangular coordinates as described above in this example. Also, in the case of rectangular coordinates, the origin of the of the coordinates is transformed so as to be translated to the origin O as shown in FIG. 3 and so on as necessary. Next, the ECU 120 corrects the coordinates of the target detected by the radar 16 (S505). Finally, the ECU 120 updates the positional information regarding the target with the corrected coordinates (S507), and terminates processing” at paragraph 0046, fourth to last sentence); and
based on the obtained rotation value, identify at least one object regarding the video information (“The position of the preceding vehicle thus corrected is updated at predetermined time intervals, e.g. time intervals in the range of several tens of milliseconds to 100 milliseconds. The identity of the object is determined based on the closeness and predicted movement of the detected position at each update, or an image captured by the camera 17, the preceding tracking target vehicle is identified, and the position thereof is specified” at paragraph 0054, line 1).
Kaita et al. does not explicitly disclose that the image information is video information, based on obtaining the video information, obtain, second rotation data by using a designated estimation model and based on the first rotation data and the second rotation data, obtain a rotation value for the video information.
Lim et al. teaches a method of an electronic device in the same field of endeavor of motorcycle tilt analysis, comprising:
obtaining video information by using the at least one camera (“For the video recording, the video is recorded at 29.97 frames per second” at section 5, line 5);
based on obtaining the video information, obtain, first rotation data on the electronic device by using the gyro sensor (“The first method to detect the leaning angle of the ego vehicle and the assumption is that a lean to the right will correspond to the rightwards trajectory of the ego vehicle, and a lean to the left will correspond to the leftwards trajectory accordingly. To detect the leaning angle, a gyroscope and an accelerometer are required, which can be found in an IMU” at section 4.1, line 1), and second rotation data by using a designated estimation model (“Trajectory prediction is very important in the advanced FCW system because by understanding the trajectory of the detected vehicle, the FCWalgorithm will have more time to alert the ego vehicle if there will be a potential collision. The trajectory prediction input data is based on pixel center coordinates of the detected vehicle using the object detection algorithm in the previous section to obtain time-series data” at section 3, line 1; this determines the vehicle’s heading, which is characterized by direction and steering angle);
based on the first rotation data and the second rotation data, obtain a rotation value for the video information (“Similarly, the roll angle will have a threshold that classify if a bike is leaning to the left, straight or right” at section 5.4, line 8; “The image will be segmented into 3 parts which indicates left, straight or right and these regions will be used to classify where does the trajectory of the vehicle detected fall under” at section 5.4, line 5; when the two directions match, the heading is determined to correspond to the left, right or no rotation).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize image derived data as taught by Lim et al. along with the gyro data of Kaita et al. to ensure that the true heading of the vehicle is reflected in the subsequent processing.
Regarding claims 2 and 12, the Kaita et al. and Lim et al. combination discloses a device and method as described in claims 1 and 11 above.
The Kaita et al. and Lim et al. combination does not explicitly disclose that the at least one processor is further configured to: based on the first rotation data and the second rotation data, obtain the rotation value for the video information by using a complementary filter.
However, Lim et al. further teaches sensor fusion utilizing complementary filtering (“These two sensors are combined with the use of a complementary filter or Kalman filter for leaning angle detection” at section 4.1, line 9).
As such, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a complementary filter for the image and gyro data “to use each sensor to make up for the other sensor’s weaknesses” (Lim et al. at section 4.1, paragraph 1, last sentence).
Regarding claims 3 and 13, Lim et al. discloses a device and method wherein the at least one processor is further configured to:
set the video information as input data of the designated estimation model (“Trajectory prediction is very important in the advanced FCW system because by understanding the trajectory of the detected vehicle, the FCWalgorithm will have more time to alert the ego vehicle if there will be a potential collision. The trajectory prediction input data is based on pixel center coordinates of the detected vehicle using the object detection algorithm in the previous section to obtain time-series data” at section 3, line 1); and
based on output data of the designated estimation model, obtain the second rotation data (“The image will be segmented into 3 parts which indicates left, straight or right and these regions will be used to classify where does the trajectory of the vehicle detected fall under” at section 5.4, line 5).
Regarding claims 4 and 14, the Kaita et al. and Lim et al. combination discloses a device and method wherein the at least one processor is further configured to:
train the designated estimation model based on a first training video and a second training video (“For riding pattern classification, the 10 hz IMU data was used as a higher sampling rate will result in increased data inputs for LSTM, which makes it more complex. After processing with time step 40 and time shift of 20 and under-sampling, the data is trained for riding pattern classification.” Lim et al. at section 5.3.2, line 1).
The Kaita et al. and Lim et al. combination does not explicitly disclose that the first training video is rotated according to a designated rotation value.
However, it is well-known in the art to utilize training data with known classes and labels, including those with specific parameters of rotation, translation, skew, etc. Lim et al. further implies the use of labeled training data in section 5.3.2. in paragraph 2. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a particular rotation value for purposes of validating the training of the model.
Regarding claims 7 and 17, the Kaita et al. and Lim et al. combination discloses a device and method wherein the first rotation data includes a rotation value identified based on an axis corresponding to a direction toward which the at least one camera faces (“First, the ECU 120 detects the current inclination of the vehicle body using the gyro sensor 30 (501). The detected inclination includes at least a roll angle and a pitch angle. Next, the ECU 120 detects the target using a sensor, i.e. the radar 16, and detects the coordinates thereof (S503)” Kaita et al. at paragraph 0046, line 9; as the sensor is forward facing, a roll angle corresponds to a rotation relative to the axis through the camera lens).
Regarding claims 8 and 18, Kaita et al. and Lim et al. combination discloses a device and method further comprising:
a display (implied by Lim et al. that the information is displayed), and
wherein the at least one processor is further configured to:
while the first video according to the video information is displayed through the display, display, superimposed on the video, at least one bounding box for identifying the at least one object; and based on the at least one bounding box, display, superimposed on the first video, information on the at least one object (Figure 8 of Lim et al.; “As seen in Figure 8, there are a few other elements in the picture as compared to normal vehicle detection. At the top, it shows words in green that describe the direction the ego vehicle is moving. From the image, it says that the bike is moving straight. This indicates the direction of the ego vehicle. Below it, it shows a “WARNING” message in blue, and the warning signal will only trigger under the correct conditions. In the image, it has two vertical blue lines, which indicate the left, straight and right regions of the image. These regions are based on x pixel threshold, and it will indicate if a detected vehicle trajectory is within one of these regions. Toward the bottom of the image, there is a green bounding box with the numbers 2.34 on the top left of the green box. The green box indicates the bounding box” at section 5.4, paragraph 2).
Regarding claims 9 and 19, the Kaita et al. and Lim et al. combination discloses a device and method wherein the electronic device is disposed in the vehicle, and wherein the video information is obtained by using at least one camera disposed toward a front direction of the vehicle (“The camera 17 captures an image of what is in front thereof through the opening or the transparent member” Kaita et al. at paragraph 0027, last sentence; Lim et al. also discloses a smartphone installed on the motorcycle facing the front).
Claim(s) 5, 6, 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kaita et al. and Lim et al. as applied to claims 1 and 11 above, and further in view of Torii (US 20170154224).
Regarding claims 5 and 15, the Kaita et al. and Lim et al. combination discloses a device and method wherein the at least one processor is further configured to:
identify, in the video, the at least one object by using an object detection model (“The vehicle detection algorithm that is used will be using the You-Only-Look-Once detection algorithm because it has proven to be state of the art with extremely fast processing speed of up to 244 frames per second (fps) with reasonable accuracy of 78.6 mean average precision (maP) by Redmon et al. (2016). The output of the YOLO boundary box coordinates to obtain the boundary box width and height will be used for distance estimation, and the bounding box center coordinates will be used for trajectory prediction” Lim et al. at section 2, paragraph 2, line 6);
obtain a third video in which at least one bounding box for the at least one object is superimposed and displayed on the second video (see Figure 8 of Lim et al.).
The Kaita et al. and Lim et al. combination does not explicitly disclose based on the obtained rotation value, obtain a second video in which a first video is rotated according to the video information.
Torii teaches a device and method in the same field of endeavor of tilt detection for motorcycles, wherein the at least one processor is further configured to:
based on the obtained rotation value, obtain a second video in which a first video is rotated according to the video information (“When the tilt angle data and the image data are input from the imaging section 3, the image-rotation section 4 rotates the image data for the correction on the basis of the tilt angle data and generates the corrected image data” at paragraph 0032, line 1);
identify, in the second video, the at least one object by using an object detection model (‘When the corrected image data is input, the image-recognition section 5 performs image-recognition processing on the basis of the corrected image data so as to recognize the surrounding situation of the vehicle body, and outputs the position information data, which includes the position information of each of the recognized objects and the like, to the drive assist control section 6” at paragraph 0033, line 1);
obtain a third video in which at least one object is superimposed and displayed on the second video (“For example, the image that corresponds to the position information data may display the position information of each of the recognized objects and the like on the image” at paragraph 0033, last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the display correction as taught by Torii for the object video data of the Kaita et al. and Lim et al. combination so “even when the vehicle body or the road surface r is tilted, the image recognition can appropriately be performed” (Torii at paragraph 0041, last sentence).
The Kaita et al., Lim et al. and Torii combination does not explicitly disclose that
based on the third video and the obtained rotation value, obtain a fourth video in which the at least one bounding box is superimposed and displayed in the first video.
However, as the rotation value is known it is simple to apply that value to the corrected bounding box image to apply to the original uncorrected image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to do so to provide the user with the location of the detected object from a perceived perspective as they are operating the motorcycle for better understanding of the location of the obstacle.
Regarding claims 6 and 16, the Kaita et al. and Lim et al. combination discloses a device and method wherein the at least one processor is further configured to:
identify a first video according to the video information (original video);
identify, in the first video, the at least one object by using an object detection model (“The vehicle detection algorithm that is used will be using the You-Only-Look-Once detection algorithm because it has proven to be state of the art with extremely fast processing speed of up to 244 frames per second (fps) with reasonable accuracy of 78.6 mean average precision (maP) by Redmon et al. (2016). The output of the YOLO boundary box coordinates to obtain the boundary box width and height will be used for distance estimation, and the bounding box center coordinates will be used for trajectory prediction” Lim et al. at section 2, paragraph 2, line 6);
obtain a second video in which at least one bounding box for the at least one object is superimposed and displayed on the first video (see Figure 8 of Lim et al.).
The Kaita et al. and Lim et al. combination does not explicitly disclose based on the obtained rotation value, obtain a third video in which the at least one bounding box in the second video is corrected and displayed.
Torii teaches a device and method in the same field of endeavor of tilt detection for motorcycles, wherein the at least one processor is further configured to:
based on the obtained rotation value (“When the tilt angle data and the image data are input from the imaging section 3, the image-rotation section 4 rotates the image data for the correction on the basis of the tilt angle data and generates the corrected image data” at paragraph 0032, line 1), obtain a third video in which the at least one detected object in the second video is corrected and displayed (‘When the corrected image data is input, the image-recognition section 5 performs image-recognition processing on the basis of the corrected image data so as to recognize the surrounding situation of the vehicle body, and outputs the position information data, which includes the position information of each of the recognized objects and the like, to the drive assist control section 6” at paragraph 0033, line 1; “For example, the image that corresponds to the position information data may display the position information of each of the recognized objects and the like on the image” at paragraph 0033, last sentence).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the display correction as taught by Torii for the object video data of the Kaita et al. and Lim et al. combination so “even when the vehicle body or the road surface r is tilted, the image recognition can appropriately be performed” (Torii at paragraph 0041, last sentence).
Claim(s) 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Kaita et al. and Lim et al. as applied to claims 1 and 11 above, and further in view of Giraud et al. (US 11,189,166).
The Kaita et al. and Lim et al. combination discloses a device and method wherein the video information is obtained by using the at least one camera disposed toward a front direction of the user (“The camera 17 captures an image of what is in front thereof through the opening or the transparent member” Kaita et al. at paragraph 0027, last sentence; Lim et al. also discloses a smartphone installed on the motorcycle facing the front).
The Kaita et al. and Lim et al. combination does not explicitly disclose that the electronic device is configured to be wearable on a body part of a user.
Giraud et al. teaches a device and method in the same field of endeavor of motorcycle safety systems wherein the electronic device is configured to be wearable on a body part of a user (“An environmental sensor such as a camera may in some embodiments be mounted on the helmet of the rider, and be connected via a Bluetooth™ protocol to the control unit 22. Helmet mounted cameras may point forwards, sideways and backwards” at col. 5, line 9).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the helmet mounted camera as taught by Giraud et al. to gather the data for the Kaita et al. and Lim et al. combination as an alternative way to capture the video data for safety analysis.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Strickland (US 20190098953) is pertinent as disclosing a helmet-based motorcycle safety system that analyzes captured data for orientation and adjusts the data accordingly for proper analysis of collision probability. Berniolles et al. (US 20230278573) and Fukuzawa (US 12548110) also disclose motorcycle safety systems that analyzes captured data for orientation and adjusts the data accordingly for proper analysis of collision probability.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672
Prosecution Insights