DETAILED ACTIONS
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim this application being in benefit of foreign priority from Korean Patent Application No. KR10-2022-0096685 filed on August 3, 2022 and Korean Patent Application No. KR10-2023-0101557 filed on August 3, 2023.
Drawings
The 19-page drawings have been considered and placed on record in the file.
Status of Claims
Claims 1-20 are pending.
Response to Amendment
The amendment filed 01/16/2026 has been entered in full.
Response to Arguments
Applicant’s arguments with respect to claim 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 4, 6-9, 12-13, 15, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Nagata et al., (US 2020/0320309 A1, published 10/08/2020), hereinafter referred to as Nagata, in view of Lipchin et al., (US 2020/0097769 A1, published 03/26/2020), hereinafter referred to as Lipchin, .
Claim 1
Nagata discloses a method of controlling a vehicular electronic device (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”) comprising:
acquiring an image via image capturing (Nagata, Fig. 2, step 210, capture or detect external image data);
detecting an object in the image in a specific format based on deep learning (Nagata, Fig. 2, step 216, recognize objects and/or actions of object within the surrounding environment from the image data, [0046], “, the detection system 100 adaptively uses machine learning to predict an object or an action of an object that is not consistent with the surrounding environment of a location, and thus, predict that the object or an action of the object may be unusual”);
performing post-processing (Nagata, [0017], “he detection system uses artificial intelligence including machine algorithm learning with models to anticipate, predict or otherwise determine when an unusual activity occurs or is about to occur.”, [0057], “The detection system 100 may predict or determine the unusual activity using a machine learning algorithm that utilizes a baseline or a baseline model to make the prediction or determination using pattern”)
detecting, based on the post-processing (Nagata, [0017], “he detection system uses artificial intelligence including machine algorithm learning with models to anticipate, predict or otherwise determine when an unusual activity occurs or is about to occur.”), whether an event (Nagata, [0044], “the detection system 100 may obtain or generate a baseline or a baseline model of objects, coupled objects and/or motions of the object for a given location (204). The detection system 100 uses the baseline or the baseline model to predict, determine or otherwise detect unusual activities. For example, the detection system 100 may predict when a bystander may attempt to steal or otherwise break into the vehicle 102.”) for activating recoding occurs (Nagata, [0017], “The detection system uses artificial intelligence including machine algorithm learning with models to anticipate, predict or otherwise determine when an unusual activity occurs or is about to occur. By anticipating, predicting or otherwise determining when the unusual activity occurs or is about to occur, the detection system proactively anticipates the unusual activity and may act to prevent, report or otherwise record or document the unusual activity.”) and starting recoding the image if the occurrence of the event is detected (Nagata, [0007], “The electronic control unit may be configured to record and capture, in the memory and using the second camera, the second image data for the time period in response to the determination that the object is different than the baseline.”, [0025], “The ECU 108 may analyze the external and internal environment of the vehicle 102 and compare the data to a baseline and/or input the data into a model to anticipate, predict or otherwise determine any unusual activities within the environment. If an unusual activity is predicted or otherwise detected, the ECU 108 may act to record, document, provide or otherwise act to mitigate consequences of the unusual activity.”, [0027], “For example, the user interface 122 may receive user input that may include configurations as to the amount of image data or the length of the video to record when an unusual activity is detected.”).
Nagata does not explicitly disclose setting a region of interest (ROI) by configuring the Roi within the captured image, performing post-processing which determines whether a status of the detected object satisfies a predetermined condition, and wherein the predetermined condition is satisfied when a bounding box of the specific format overlaps the RoI, the bounding box surrounding the object and being configured to determine an inclusion relationship with respect to the RoI.
However, Lipchin teaches setting a region of interest (ROI) by configuring the Roi within the captured image (Lipchin, [0122], “In FIG. 10A to 10G, the boundary of the default region is illustrated in solid lines, and the boundary of the frame VF is illustrated in broken lines. In this example, there are 6 default regions DR1, DR2, DR3, DR4, DR5, and DR6. The default regions DR1-DR6 overlap each other within the frame VF as shown in FIG. 10G.”), performing post-processing (which determines whether a status of the detected object satisfies a predetermined condition (Lipchin, [0074], “Examples of determinations made by the video analytics module 224 may include one or more of foreground/background segmentation, object detection, object tracking, object classification, virtual tripwire, anomaly detection, facial detection, facial recognition, license plate recognition, identifying objects “left behind” or “removed”, unusual motion detection, appearance matching, characteristic (facet) searching, and business intelligence.”, [0007], “The outputs from the CNN may comprise bounding boxes for detected objects together with a classification score (or confidence).”, [0107], “The tracker 408 also includes a categorizer module 606 for categorizing each predicted target based on the matching results. The categories assigned by the categorizer module 606 are referred to as status categories. These status categories form a part of the tracker feedback 410. In one embodiment, the status categories may indicate the priority of targets based on the target's need to be detected. In one embodiment, the status category assigned to a target may indicate how long since the target has been matched or detected.”, [0117], “the inputs 420 include the motion vectors, the predefined regions (“RPs”) generated from the object size map, and the LCT map”, [0122], “In FIG. 10A to 10G, the boundary of the default region is illustrated in solid lines, and the boundary of the frame VF is illustrated in broken lines. In this example, there are 6 default regions DR1, DR2, DR3, DR4, DR5, and DR6. The default regions DR1-DR6 overlap each other within the frame VF as shown in FIG. 10G. In FIG. 10G, all of the boundaries of the default regions are shown for illustration purpose. These default regions are created in a sliding windows manner. For example, the default regions could be 2 by 3 partially overlapped bounding boxes windows over the whole frame as shown in FIGS. 10A to 100 and FIGS. 10D to 10F.”, [0125], “At step 914, to prepare a subset of candidate regions (denoted as “RP_2”), it is determined whether each of the predicted targets are included in any region of the regions in subset RP_1. It is determined how much of the bounding box of the predicted target intersects with each of the regions in subset RP_1. For example, if the intersection area between the bounding box of a predicted target and a region is greater than a certain percentage K of a target area (i.e., the area of the bounding box) for a predicted target, the system determines that the predicted target is within the region, i.e. the target belongs to that region. In one embodiment, K may be 80%.”, [0128], “At step 918, a region priority score of each region in the candidate regions subset RP_2 is calculated. In one embodiment, using the predicted target(s) inside the corresponding region, a region priority score of that region is generated using the priority score of each predicted target. For example, if a target with the invisible target 810 category is considered as to be inside Region R1, the priority score S1 is added to the region priority score of Region R1.”), and wherein the predetermined condition (Lipchin, [0135], “The target prediction intersection score S11 represents the degree of overlap of a region proposal with target prediction bounding boxes. Score S11 may be computed from the tracker feedback 410. A larger proportion of overlap area results in a higher score S11. In one embodiment, the overlap area may undergo a nonlinear scaling to produce S11. In one embodiment, the score S11 may depend on the motion status category from the tracker 408. The motion status categories include moving and stopped.”) is satisfied when a bounding box of the specific format overlaps the RoI, the bounding box surrounding the object and being configured to determine an inclusion relationship with respect to the RoI (Lipchin, [0117], “the inputs 420 include the motion vectors, the predefined regions (“RPs”) generated from the object size map, and the LCT map”, [0122], “In FIG. 10A to 10G, the boundary of the default region is illustrated in solid lines, and the boundary of the frame VF is illustrated in broken lines. In this example, there are 6 default regions DR1, DR2, DR3, DR4, DR5, and DR6. The default regions DR1-DR6 overlap each other within the frame VF as shown in FIG. 10G. In FIG. 10G, all of the boundaries of the default regions are shown for illustration purpose. These default regions are created in a sliding windows manner. For example, the default regions could be 2 by 3 partially overlapped bounding boxes windows over the whole frame as shown in FIGS. 10A to 100 and FIGS. 10D to 10F.”, [0125], “At step 914, to prepare a subset of candidate regions (denoted as “RP_2”), it is determined whether each of the predicted targets are included in any region of the regions in subset RP_1. It is determined how much of the bounding box of the predicted target intersects with each of the regions in subset RP_1. For example, if the intersection area between the bounding box of a predicted target and a region is greater than a certain percentage K of a target area (i.e., the area of the bounding box) for a predicted target, the system determines that the predicted target is within the region, i.e. the target belongs to that region. In one embodiment, K may be 80%.”,[0128], “At step 918, a region priority score of each region in the candidate regions subset RP_2 is calculated. In one embodiment, using the predicted target(s) inside the corresponding region, a region priority score of that region is generated using the priority score of each predicted target. For example, if a target with the invisible target 810 category is considered as to be inside Region R1, the priority score S1 is added to the region priority score of Region R1.”).
Nagata and Lipchin are both considered to be analogous to the claimed invention because they are in the same field of unusual activity detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Nagata to incorporate the teachings of Lipchin of setting a region of interest (ROI) by configuring the Roi within the captured image, performing post-processing which determines whether a status of the detected object satisfies a predetermined condition, and wherein the predetermined condition is satisfied when a bounding box of the specific format overlaps the RoI, the bounding box surrounding the object and being configured to determine an inclusion relationship with respect to the RoI. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to improve object detection performance and reach higher detection accuracy (Lipchin, [0098]).
Claim 2
The combination of Nagata in view of Lipchin discloses the method of claim 1 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”), further comprising: determining that the occurrence of the event has been detected when a degree of overlap between the object and the RoI exceeds a preset level (Lipchin, [0125], “At step 914, to prepare a subset of candidate regions (denoted as “RP_2”), it is determined whether each of the predicted targets are included in any region of the regions in subset RP_1. It is determined how much of the bounding box of the predicted target intersects with each of the regions in subset RP_1. For example, if the intersection area between the bounding box of a predicted target and a region is greater than a certain percentage K of a target area (i.e., the area of the bounding box) for a predicted target, the system determines that the predicted target is within the region, i.e. the target belongs to that region. In one embodiment, K may be 80%.”. The proposed combination as well as the motivation for combining the Nagata and Lipchin references presented in the rejection of Claim 1, apply to Claim 2 and are incorporated herein by reference. Thus, the method recited in Claim 2 is met by Nagata and Lipchin.
Claim 4
The combination of Nagata in view of Lipchin discloses the method of claim 1 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”),, wherein the inclusion relationship is a relationship where at least a portion of the bounding box is included in the RoI (Lipchin, [0117], “the inputs 420 include the motion vectors, the predefined regions (“RPs”) generated from the object size map, and the LCT map”, [0122], “In FIG. 10A to 10G, the boundary of the default region is illustrated in solid lines, and the boundary of the frame VF is illustrated in broken lines. In this example, there are 6 default regions DR1, DR2, DR3, DR4, DR5, and DR6. The default regions DR1-DR6 overlap each other within the frame VF as shown in FIG. 10G. In FIG. 10G, all of the boundaries of the default regions are shown for illustration purpose. These default regions are created in a sliding windows manner. For example, the default regions could be 2 by 3 partially overlapped bounding boxes windows over the whole frame as shown in FIGS. 10A to 100 and FIGS. 10D to 10F.”, [0125], “At step 914, to prepare a subset of candidate regions (denoted as “RP_2”), it is determined whether each of the predicted targets are included in any region of the regions in subset RP_1. It is determined how much of the bounding box of the predicted target intersects with each of the regions in subset RP_1. For example, if the intersection area between the bounding box of a predicted target and a region is greater than a certain percentage K of a target area (i.e., the area of the bounding box) for a predicted target, the system determines that the predicted target is within the region, i.e. the target belongs to that region. In one embodiment, K may be 80%.”,[0128], “At step 918, a region priority score of each region in the candidate regions subset RP_2 is calculated. In one embodiment, using the predicted target(s) inside the corresponding region, a region priority score of that region is generated using the priority score of each predicted target. For example, if a target with the invisible target 810 category is considered as to be inside Region R1, the priority score S1 is added to the region priority score of Region R1.”). The proposed combination as well as the motivation for combining the Nagata and Lipchin references presented in the rejection of Claim 1, apply to Claim 4 and are incorporated herein by reference. Thus, the method recited in Claim 4 is met by Nagata and Lipchin.
Claim 6
The combination of Nagata in view of Lipchin discloses the method of claim 1 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”), wherein the predetermined condition includes a case where an intersection over union (IoU) between the bounding box and the RoI configured in the image is equal to or larger than a preset value (Lipchin, [0125], “At step 914, to prepare a subset of candidate regions (denoted as “RP_2”), it is determined whether each of the predicted targets are included in any region of the regions in subset RP_1. It is determined how much of the bounding box of the predicted target intersects with each of the regions in subset RP_1. For example, if the intersection area between the bounding box of a predicted target and a region is greater than a certain percentage K of a target area (i.e., the area of the bounding box) for a predicted target, the system determines that the predicted target is within the region, i.e. the target belongs to that region. In one embodiment, K may be 80%.”). The proposed combination as well as the motivation for combining the Nagata and Lipchin references presented in the rejection of Claim 1, apply to Claim 6 and are incorporated herein by reference. Thus, the method recited in Claim 6 is met by Nagata and Lipchin.
Claim 7
The combination of Nagata in view of Lipchin discloses the method of claim 1 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”), wherein the specific format includes a skeleton format which represents a pose of the object as a simplified skeleton shape (Nagata, [0017], “The detection system uses artificial intelligence including machine algorithm learning with models to anticipate, predict or otherwise determine when an unusual activity occurs or is about to occur.”, [0053], “The detection system 100 recognizes objects and actions of the objects from the image data (216). In order to recognize the different objects and actions of the objects within the image data, the detection system 100 may segment, outline or otherwise map figures within the image data using multiple joints and segments. The segments may represent linear representations or outlines of an object and the joints may represent vertices, contours or other angles between the different segments.”, [0054], “Once the outline of the object is mapped, the detection system 100 may compare the representation of the multiple joints and segments to a database of objects, which have already been mapped, to identify the object. For example, the detection system 100 may compare an outline of a person within the surrounding environment to a stored outline of a person and determine that the shape of the outlines match, and so the detection system 100 recognizes the object as a person”, a skeleton shape is made of different joints and segments between joints).
Claim 8
The combination of Nagata in view of Lipchin discloses the method of claim 7 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”), wherein the predetermined condition is when similarity between a skeleton shape of the object and a behavior pattern is equal to or larger than a preset value (Nagata, [0054], “Once the outline of the object is mapped, the detection system 100 may compare the representation of the multiple joints and segments to a database of objects, which have already been mapped, to identify the object. For example, the detection system 100 may compare an outline of a person within the surrounding environment to a stored outline of a person and determine that the shape of the outlines match, and so the detection system 100 recognizes the object as a person..”, [0056], “The detection system may outline the shape of these objects using segments and joints to represent the linear representations and the changes in the angle of the linear representations, respectively, and then compare the outlined shape to different outline shapes of different objects or combination of objects in a database. When the outline of the object matches or is within a threshold error of one of the outlined shapes in the database, the detection system 100 may determine that the outlined object is the matched object in the database.”, [0057], “The detection system 100 may predict or determine the unusual activity using a machine learning algorithm that utilizes a baseline or a baseline model to make the prediction or determination using patterns. An object is unusual when the object does not belong within the surrounding environment, such as when the object does not usually appear in the surrounding environment. The detection system 100 may determine that the object does not usually appear in the surrounding environment when the object that is recognized is not part of the baseline of the surrounding environment. An action of the object is unusual when the motion of the object presents a threat to the vehicle 102, such as when the object may collide with the vehicle 102, or when the action of the object is not consistent with a regular motion of the object within the baseline of the surrounding environment.”).
Claim 9
The combination of Nagata in view of Lipchin discloses the method of claim 1 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”), further comprising: transmitting the recorded image (Nagata, [0007], “The electronic control unit may be configured to record and capture, in the memory and using the second camera, the second image data for the time period in response to the determination that the object is different than the baseline.”) to a user terminal of an owner of a vehicle in which the vehicular electronic device is installed (Nagata, [0035], “The communication device may notify or provide documentation to the owner of the vehicle 102 that an unusual activity in proximity to the vehicle 102 is about to occur, has occurred or is occurring”), wherein the recorded image is transmitted in response to a determination that the predetermined condition is satisfied (Nagata, [0062], “The recorded information may allow for a claims adjuster or owner of the vehicle 102 to identify any lost or stolen items from within the vehicle 102. “, the recorded video has to be transmitted to the owner for them to use it, [0064], “The detection system 100 may operate or control one or more vehicle components in response to the detecting the object that is unusual and/or action of the object that is unusual (224). The detection system 100 may alert and notify a third-party, such as the owner of the vehicle 102, the police or a security service provider when the unusual object or action is detected, for example. The detection system 100 may perform other actions, such as disable the ignition, lock the doors, activate a GPS device to provide location information of the vehicle 102, close the windows, or activate an audio and/or visual alert to prevent an unidentified person from starting or gaining access to the vehicle 102.”, Lipchin teaches the predetermine condition being satisfied and Nagata teaches detecting the event).
Claims 12-13, 15, and 17-18 are rejected for similar reasons as those described in claims 1-2, 3, and 6-8, respectively. The additional elements in Claims 12-13, 15, and 17-18 (Nagata and Lipchin) discloses includes: a vehicular electronic device (Nagata, Fig. 1) comprising: a camera unit installed in a vehicle (Nagata, Fig. 1, external cameras 116a) and configured to acquire an image of surroundings of the vehicle (Nagata, [0005], “The multiple external cameras may be configured to capture different views of the surrounding environment.”), a processor (Nagata, [0022],” The detection system 100 includes one or more processors, such as an electronic control unit (ECU) 108 and a memory 110 ), and a storage unit (Nagata, [0022],” The detection system 100 includes one or more processors, such as an electronic control unit (ECU) 108 and a memory 110 ). The proposed combination as well as the motivation for combining the Nagata and Lipchin references presented in the rejection of Claim 1, apply to Claims 12-13, 15, and 17-18 and are incorporated herein by reference. Thus, the device recited in Claim 12-13, 15, and 17-18 is met by Nagata and Lipchin.
Claims 5 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Nagata in view of Lipchin in view of Hoang et al., (US 2023/0351761 A1, earliest filing data of foreign application is 04/28/2022), hereinafter referred to as Hoang.
Claim 5
The combination of Nagata in view of Lipchin discloses the method of claim 4 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”).
The combination of Nagata in view of Lipchin does not explicitly disclose wherein the at least a portion of the bounding box includes a lower boundary of the bounding box.
However, Hoang teaches wherein the at least a portion of the bounding box includes a lower boundary of the bounding box (Hoang, Fig. 2, the WOI is analogous to the region of interest and the lower boundary of the bounding boxes 18, 20, 21, 22, and 19 are included, [0038], “A preferably filtering condition parameter includes the threshold for the ratio value in regard of the overlap of the boundary box and the window of interest.”).
Nagata, Lipchin, and Hoang are all considered to be analogous to the claimed invention because they are in the same field of image processing of vehicular images. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Nagata in view of Lipchin to incorporate the teachings of Hoang of wherein the at least a portion of the bounding box includes a lower boundary of the bounding box. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to avoid reducing the tracking accuracy and reliability (Hoang, [0024]).
Claim 16 is rejected for similar reasons as those described in claim 5. The additional elements in Claim 16 (Nagata, Lipchin, and Hoang) discloses includes: a vehicular electronic device (Nagata, Fig. 1) comprising: a camera unit installed in a vehicle (Nagata, Fig. 1, external cameras 116a) and configured to acquire an image of surroundings of the vehicle (Nagata, [0005], “The multiple external cameras may be configured to capture different views of the surrounding environment.”), a processor (Nagata, [0022],” The detection system 100 includes one or more processors, such as an electronic control unit (ECU) 108 and a memory 110 ), and a storage unit (Nagata, [0022],” The detection system 100 includes one or more processors, such as an electronic control unit (ECU) 108 and a memory 110 ). The proposed combination as well as the motivation for combining the Nagata, Lipchin, and Hoang references presented in the rejection of Claim 5, apply to Claim 16 and are incorporated herein by reference. Thus, the device recited in Claim 16 is met by Nagata, Lipchin, and Hoang
Claims 10-11 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Nagata in view of Lipchin in further view of Fu et al., (US 11400958 B1, published 08/02/2022), hereinafter referred to as Fu.
Claim 10
The combination of Nagata in view of Lipchin discloses the method of claim 9 (Nagata, Abstract, “Methods, systems, and apparatus for a detection system. The detection system includes a first camera configured to capture a first image data of a surrounding environment. The first image data includes multiple objects. The detection system includes a memory configured to store image data. The detection system includes an electronic control unit. The electronic control unit is configured to obtain the first image data. The electronic control unit is configured to recognize the multiple objects. The electronic control unit is configured to determine that an object among the multiple objects within the surrounding environment is different than a baseline of the surrounding environment. The electronic control unit is configured to record and capture, in the memory and using the camera, the first image data for a time period before and after the determination that the object is different than the baseline.”).
The combination of Nagata in view of Lipchin does not explicitly disclose receiving an object detection model and an event detection model, wherein the object detection model and the event detection model are reinforced based on evaluation data regarding a level of satisfaction with the recorded image, the evaluation data generated at the user terminal.
However, Fu teaches receiving an object detection model (Fu, Abstract, “The information includes a state of a vehicle and a state of an agent in the vehicle's environment. The first state information is processed with a neural network to determine at least one action to be performed by the agent, including a perception degradation action causing misperception of the agent by a perception system of the vehicle. “, Col. 13, lines 15-26, “perception system 402 receives data associated with at least one physical object (e.g., data that is used by perception system 402 to detect the at least one physical object) in an environment and classifies the at least one physical object. In some examples, perception system 402 receives image data captured by at least one camera (e.g., cameras 202a), the image associated with (e.g., representing) one or more physical objects within a field of view of the at least one camera. In such an example, perception system 402 classifies at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, pedestrians, and/or the like).”) and an event detection model (Fu, Col. 19, lines 54-58, “In some embodiments, actions of agents can include normal behavioral actions (such as moving in a particular direction and at a particular speed) and perception degradation (e.g., performing an action not yet considered by the perception system)”, Col. 21, lines 59-65, “the first state information (e.g., a speed and direction at the first location 516) is processed with a neural network by the at least one processor to determine at least one action to be performed by the agent. In some embodiments, the action of the agent is a normal action, such as move forward, backward, left, or right with a specific velocity and/or a perception degradation action”)), wherein the object detection model and the event detection model are reinforced based on evaluation data regarding a level of satisfaction with the recorded image, the evaluation data generated at the user terminal (Fu, Col. 3, lines 46-56, “A reward for the at least one action is determined by the at least one processor, including the following. A first distance between the vehicle and the agent in the first state is determined based on the first state information. A second distance between the vehicle and the agent in the second state is determined based on the second state information. The first distance and the second distance are compared to determine the reward for the at least one action. The reward is greater when the second distance meets the first distance. At least one weight of the neural network is adjusted by the at least one processor based on the reward for the at least one action.”, Col. 19, lines 46-57, “The edge case scenarios can be used as additions to the CNN 440, for example. Identifying the edge case scenarios can include using a reinforcement learning process to identify a right policy, e.g., how the AV should react safely to navigate through a situation, such as an action (e.g., moving, or even being present) by an agent. In some embodiments, perception degradation is one of the agent's possible actions that can allow an agent to collide with the AV. In some embodiments, actions of agents can include normal behavioral actions (such as moving in a particular direction and at a particular speed) and perception degradation (e.g., performing an action not yet considered by the perception system).”).
Nagata, Lipchin, and Fu are all considered to be analogous to the claimed invention because they are in the same field of image processing of vehicular images. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Nagata in view of Lipchin to incorporate the teachings of Fu of receiving an object detection model and an event detection model, wherein the object detection model and the event detection model are reinforced based on evaluation data regarding a level of satisfaction with the recorded image, the evaluation data generated at the user terminal. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been because it can lead to more accurate training of machine learning models involved in the operation of an autonomous vehicle (Fu, Col. 4, lines 7-9).
Claim 11
The combination of Nagata in view of Lipchin in view of Fu discloses the method of claim 10 (Ban, [0002], “The disclosure relates to an electronic device and method of controlling an operation of a vehicle, and for example, to an electronic device and a method of providing a notification message notifying about an occurrence of an event related to driving of a vehicle according to position of an object in a plurality of frames.”), further comprising: updating the object detection model and the event detection model based on reinforced machine learning technology by using the evaluation data (Fu, Col. 22, lines 41-46, “the reward engine 624 updates reward information on different edges of the neural network based on the action taken by the AV 506 in response to the jaywalker 508. In some embodiments, determining a reward for the at least one action includes steps 710-714”, Col. 19, lines 43-52, “a goal of updating the CNN 440 can include constructing edge case scenarios from the viewpoint of other agents (for example, pedestrians, bicycles, motorcycles, or another vehicles). The edge case scenarios can be used as additions to the CNN 440, for example. Identifying the edge case scenarios can include using a reinforcement learning process to identify a right policy, e.g., how the AV should react safely to navigate through a situation, such as an action (e.g., moving, or even being present) by an agent”).
The proposed combination as well as the motivation for combining the Nagata, Lipchin, and Fu references presented in the rejection of Claim 10, apply to Claim 11 and are incorporated herein by reference. Thus, the method recited in Claim 11 is met by Nagata, Lipchin, and Fu.
Claim 19
Nagata discloses a vehicle service system (Nagata, Fig. 1) comprising:
an electronic device installed in a vehicle and configured to (Nagata, Fig. 1):
acquire an image via image capturing (Nagata, Fig. 2, step 210, capture or detect external image data);
detect an object in the image in a specific format based on deep learning (Nagata, Fig. 2, step 216, recognize objects and/or actions of object within the surrounding environment from the image data, [0046], “, the detection system 100 adaptively uses machine learning to predict an object or an action of an object that is not consistent with the surrounding environment of a location, and thus, predict that the object or an action of the object may be unusual”);
detecting whether a recording event occurs (Nagata, [0044], “the detection system 100 may obtain or generate a baseline or a baseline model of objects, coupled objects and/or motions of the object for a given location (204). The detection system 100 uses the baseline or the baseline model to predict, determine or otherwise detect unusual activities. For example, the detection system 100 may predict when a bystander may attempt to steal or otherwise break into the vehicle 102.”, [0017], “The detection system uses artificial intelligence including machine algorithm learning with models to anticipate, predict or otherwise determine when an unusual activity occurs or is about to occur. By anticipating, predicting or otherwise determining when the unusual activity occurs or is about to occur, the detection system proactively anticipates the unusual activity and may act to prevent, report or otherwise record or document the unusual activity.”) based in a post-processing for the detected object (Nagata, [0017], “he detection system uses artificial intelligence including machine algorithm learning with models to anticipate, predict or otherwise determine when an unusual activity occurs or is about to occur.”, [0057], “The detection system 100 may predict or determine the unusual activity using a machine learning algorithm that utilizes a baseline or a baseline model to make the prediction or determination using pattern”),
starting recoding the image if the occurrence of the event is detected (Nagata, [0007], “The electronic control unit may be configured to record and capture, in the memory and using the second camera, the second image data for the time period in response to the determination that the object is different than the baseline.”, [0025], “The ECU 108 may analyze the external and internal environment of the vehicle 102 and compare the data to a baseline and/or input the data into a model to anticipate, predict or otherwise determine any unusual activities within the environment. If an unusual activity is predicted or otherwise detected, the ECU 108 may act to record, document, provide or otherwise act to mitigate consequences of the unusual activity.”, [0027], “For example, the user interface 122 may receive user input that may include configurations as to the amount of image data or the length of the video to record when an unusual activity is detected.”),
transmit the recorded image (Nagata, [0035], “The communication device may notify or provide documentation to the owner of the vehicle 102 that an unusual activity in proximity to the vehicle 102 is about to occur, has occurred or is occurring”); and
a user terminal configured to receive the recorded image from the electronic device (Nagata, [0064], “The detection system 100 may alert and notify a third-party, such as the owner of the vehicle 102, the police or a security service provider when the unusual object or action is detected, for example.”, [0040], “he external database 104 may include a third-party server or website that stores or provides information. The information may include real-time information, periodically updated information, or user-inputted information. A server may be a computer in a network that is used to provide services, such as accessing files or sharing peripherals, to other computers in the network”),
Nagata does not explicitly disclose setting a region of interest (ROI) by configuring the Roi within the captured image, and wherein the predetermined condition is satisfied when a bounding box of the specific format overlaps the RoI, the bounding box surrounding the object and being configured to determine an inclusion relationship with respect to the RoI.
However, Lipchin teaches setting a region of interest (ROI) by configuring the Roi within the captured image (Lipchin, [0122], “In FIG. 10A to 10G, the boundary of the default region is illustrated in solid lines, and the boundary of the frame VF is illustrated in broken lines. In this example, there are 6 default regions DR1, DR2, DR3, DR4, DR5, and DR6. The default regions DR1-DR6 overlap each other within the frame VF as shown in FIG. 10G) and wherein the predetermined condition (Lipchin, [0135], “The target prediction intersection score S11 represents the degree of overlap of a region proposal with target prediction bounding boxes. Score S11 may be computed from the tracker feedback 410. A larger proportion of overlap area results in a higher score S11. In one embodiment, the overlap area may undergo a nonlinear scaling to produce S11. In one embodiment, the score S11 may depend on the motion status category from the tracker 408. The motion status categories include moving and stopped.”) is satisfied when a bounding box of the specific format overlaps the RoI, the bounding box surrounding the object and being configured to determine an inclusion relationship with respect to the RoI (Lipchin, [0117], “the inputs 420 include the motion vectors, the predefined regions (“RPs”) generated from the object size map, and the LCT map”, [0122], “In FIG. 10A to 10G, the boundary of the default region is illustrated in solid lines, and the boundary of the frame VF is illustrated in broken lines. In this example, there are 6 default regions DR1, DR2, DR3, DR4, DR5, and DR6. The default regions DR1-DR6 overlap each other within the frame VF as shown in FIG. 10G. In FIG. 10G, all of the boundaries of the default regions are shown for illustration purpose. These default regions are created in a sliding windows manner. For example, the default regions could be 2 by 3 partially overlapped bounding boxes windows over the whole frame as shown in FIGS. 10A to 100 and FIGS. 10D to 10F.”, [0125], “At step 914, to prepare a subset of candidate regions (denoted as “RP_2”), it is determined whether each of the predicted targets are included in any region of the regions in subset RP_1. It is determined how much of the bounding box of the predicted target intersects with each of the regions in subset RP_1. For example, if the intersection area between the bounding box of a predicted target and a region is greater than a certain percentage K of a target area (i.e., the area of the bounding box) for a predicted target, the system determines that the predicted target is within the region, i.e. the target belongs to that region. In one embodiment, K may be 80%.”,[0128], “At step 918, a region priority score of each region in the candidate regions subset RP_2 is calculated. In one embodiment, using the predicted target(s) inside the corresponding region, a region priority score of that region is generated using the priority score of each predicted target. For example, if a target with the invisible target 810 category is considered as to be inside Region R1, the priority score S1 is added to the region priority score of Region R1.”).
Nagata and Lipchin are both considered to be analogous to the claimed invention because they are in the same field of unusual activity detection. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Nagata to incorporate the teachings of Lipchin of setting a region of interest (ROI) by configuring the Roi within the captured image and wherein the predetermined condition is satisfied when a bounding box of the specific format overlaps the RoI, the bounding box surrounding the object and being configured to determine an inclusion relationship with respect to the RoI. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been to improve object detection performance and reach higher detection accuracy (Lipchin, [0098]).
The combination of Nagata in view of Lipchin does not explicitly disclose generate evaluation data based on evaluation score which is input for the recorded image and a vehicle service providing server configured to receive the recorded image and the evaluation data, and generate labeling data used to reinforce a machine learning model.
However, Fu teaches generate evaluation data based on evaluation score which is input for the recorded image (Fu, Col. 2, lines 52-57, “The first distance and the second distance are compared to determine the reward for the at least one action. The reward is greater when the second distance meets the first distance. At least one weight of the neural network is adjusted by the at least one processor based on the reward for the at least one action.”, the reward is the evaluation score) and a vehicle service providing server configured to receive the recorded image and the evaluation data, and generate labeling data used to reinforce a machine learning model data (Fu, Col. 22, lines 41-46, “the reward engine 624 updates reward information on different edges of the neural network based on the action taken by the AV 506 in response to the jaywalker 508. In some embodiments, determining a reward for the at least one action includes steps 710-714”, Col. 19, lines 43-52, “a goal of updating the CNN 440 can include constructing edge case scenarios from the viewpoint of other agents (for example, pedestrians, bicycles, motorcycles, or another vehicles). The edge case scenarios can be used as additions to the CNN 440, for example. Identifying the edge case scenarios can include using a reinforcement learning process to identify a right policy, e.g., how the AV should react safely to navigate through a situation, such as an action (e.g., moving, or even being present) by an agent”).
Nagata, Lipchin, and Fu are all considered to be analogous to the claimed invention because they are in the same field of image processing of vehicular images. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method as taught by Nagata in view of Lipchin to incorporate the teachings of Fu of generate evaluation data based on evaluation score which is input for the recorded image and a vehicle service providing server configured to receive the recorded image and the evaluation data, and generate labeling data used to reinforce a machine learning model. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. The motivation for the proposed modification would have been because it can lead to more accurate training of machine learning models involved in the operation of an autonomous vehicle (Fu, Col. 4, lines 7-9).
Claim 20
The combination of Nagata in view of Lipchin in view of Fu discloses the vehicle service system of claim 19 (Nagata, Fig. 1), wherein the vehicle service providing server generates an object detection model (Fu, Abstract, “The information includes a state of a vehicle and a state of an agent in the vehicle's environment. The first state information is processed with a neural network to determine at least one action to be performed by the agent, including a perception degradation action causing misperception of the agent by a perception system of the vehicle. “, Col. 13, lines 15-26, “perception system 402 receives data associated with at least one physical object (e.g., data that is used by perception system 402 to detect the at least one physical object) in an environment and classifies the at least one physical object. In some examples, perception system 402 receives image data captured by at least one camera (e.g., cameras 202a), the image associated with (e.g., representing) one or more physical objects within a field of view of the at least one camera. In such an example, perception system 402 classifies at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, pedestrians, and/or the like).”) and an event detection model (Fu, Col. 19, lines 54-58, “In some embodiments, actions of agents can include normal behavioral actions (such as moving in a particular direction and at a particular speed) and perception degradation (e.g., performing an action not yet considered by the perception system)”, Col. 21, lines 59-65, “the first state information (e.g., a speed and direction at the first location 516) is processed with a neural network by the at least one processor to determine at least one action to be performed by the agent. In some embodiments, the action of the agent is a normal action, such as move forward, backward, left, or right with a specific velocity and/or a perception degradation action”)) by processing a machine learning technology (Fu, Col. 14, lines 46-52, “perception system 402, planning system 404, localization system 406, and/or control system 408 implement at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one auto-encoder, at least one transformer, and/or the like).”), analyzes correlation between a behavior pattern of an object in the recorded image and the evaluation data (Fu, Col. 22, lines 64-67, Col. 23, lines 1 -12, “determining the reward for the at least one action includes determining similarity metrics for actions. For example, in some embodiments, information is received that is indicative of a previous action taken by the agent in the first state during a previous execution of the driving scenario. In some embodiments, the at least one action by the agent is compared with the previous action to determine a dissimilarity metric between the at least one action and the previous action. In some embodiments, the reward is determined based on the dissimilarity metric, where the reward is greater when the dissimilarity metric indicates that there is a greater dissimilarity between the at least one action and the previous action. For example, higher rewards are given to actions (or sets of actions) with a greater dissimilarity relative to other solutions found for the same driving scenario.”), and acquires a solution for the object detection model and the event detection model (Fu, Col. 23, lines 8-11, “For example, higher rewards are given to actions (or sets of actions) with a greater dissimilarity relative to other solutions found for the same driving scenario.”, Col. 23, lines 31-54, “the least one weight of the neural network is adjusted by the at least one processor based on the reward for the at least one action. For example, in some embodiments, the autonomous system 202 represents possible actions that the agent takes in various states represented by the CNN. In some embodiments, determining the reward for the at least one action includes distance-based comparisons and adjustments. In some embodiments, information is received that is indicative of a lowest recorded distance between the vehicle and the agent during a previous execution of the driving scenario. In some embodiments, the second distance is compared with the lowest recorded distance for the driving scenario. In some embodiments, the reward for the at least one action is increased in response to a determination that the second distance is less than the lowest recorded distance. In some embodiments, the reward for each action by the agent in the driving scenario is increased before the at least one action in response to determining that the second distance is less than the lowest recorded distance. For example, if the overall distance between the agent and the vehicle is lower than the recorded lowest distance from previous executions of the driving scenario, then the reward is increased for all actions taken.”).
The proposed combination as well as the motivation for combining the Nagata, Lipchin, and Fu references presented in the rejection of Claim 19, apply to Claim 20 and are incorporated herein by reference. Thus, the system recited in Claim 20 is met by Nagata, Lipchin, and Fu.
Allowable Subject Matter
Claims 3 and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The claimed features such as “wherein the specific format includes a skeleton format which represents a skeleton shape of the object according to a pose of the object, and wherein the occurrence of the event is detected by determining that the skeleton shape of the object is similar to a specific behavior pattern and that a lower part of the bounding box has entered the RoI” claimed in dependent claims 3 and 14, in combination with the remainder of the limitations of the claims, are neither anticipated nor obvious in view of the prior art of record.
In the closest prior art of record, Nagata et al., (US 2020/0320309 A1, published 10/08/2020) teaches wherein the object detected is in the form of joints and segments to represent a shape of a person. A skeleton shape is made of plurality of joints and segments connecting the joints. Nagata also teaches using machine learning by determining a pattern to detect an unusual activity.
However, Nagata fails to teach detecting the occurrence of the event or unusual activity by comparing the skeleton shape to a pattern and that the lower part of the bounding box of the person or skeleton is inside a region of interest. The secondary reference Lipchin, which is used to teach the overlapping of the bounding box and region of interest also does not teach detecting an event by determining the skeleton shape and the existence of the lower part of the bounding box inside the region of interest. Therefore claims 3 and 14 would be allowable for claiming the limitation “wherein the specific format includes a skeleton format which represents a skeleton shape of the object according to a pose of the object, and wherein the occurrence of the event is detected by determining that the skeleton shape of the object is similar to a specific behavior pattern and that a lower part of the bounding box has entered the RoI”, in combination with the remainder of the limitations of the claims.
Because the cited prior art of records does not teach or suggest each and every feature of dependent claims 3 and 14, these claims would be allowable.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DENISE G ALFONSO whose telephone number is (571)272-1360. The examiner can normally be reached Monday - Friday 7:30 - 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DENISE G ALFONSO/
Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights