DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 6, 7, 11, 16 and 17 are objected to because of the following informalities:
Claim 6: replace coma “,” at the end of first limitation before “and” with a semicolon “;”.
Claim 7: replace coma “,” at the end of first limitation before “and” with a semicolon “;”.
Claim 11: add “and” at the end of second limitation.
Claim 16: replace coma “,” at the end of first limitation before “and” with a semicolon “;”.
Claim 17: replace coma “,” at the end of first limitation before “and” with a semicolon “;”.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a model learning part configured to…”
“a dataset calculation part configured to…” in claim 1
“a correction part configured to…” in claim 2
“a keypoint learning part configured to…” in claim 3
“a keypoint detection part configured to…” in claim 4
“a spatial coordinate calculation part configured to…” in claim 5
“a second calculation part configured to …” in claim 7
“a data generation part configured to…” in claim 8
“a setting part configured to…”
“a labelling part configured to…” in claim 9.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. Paragraph [0042] of the specifications recites that the parts refer to a unit of processing at least one function or act. For example, the terms may refer to at least process processed by at least one hardware such as FPGA/ASIC, software stored in memories or processors.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 5, 8 – 15 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (US 20220300738 A1; hereafter referred to as Hu) in view of Reddy et al. (Reddy, N. D., Vo, M., & Narasimhan, S. G. (2019). Occlusion-net: 2d/3d occluded keypoint localization using graph networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7326-7335); hereafter referred to as Reddy).
Regarding Claim 1, Hu teaches:
A vehicle location calculation apparatus (Hu, [0053] “After training, the image recognition module is able to determine the position and label of the objects and recognize the objects, a vehicle, a human etc. from an input image”), comprising:
a model learning part configured to perform learning to output an invisible keypoint set in a model image in which each vehicle is modeled, based on a visible keypoint set in the model image (Hu, [0010] “the method may include estimating occluded points in at least one of the plurality of 2D images by defining lines starting from visible marked points”; Hu, [0040] “FIG. 4A represents a 2D image 50 showing vehicle 52 in a front perspective view. The key points 54 on the vehicle image 52 are manually marked incrementally using the image feature extraction module 40...The AR SLAM engine 28 maps the key points 54 to 3D coordinate system 58. FIG. 4B represents an additional 2D image 60 showing the vehicle 52 in a rear perspective view”; Hu, [0044] “image feature extraction module 40 estimates essential occluded key points by defining lines starting from known key points”)
However, Hu fails to explicitly teach:
a model image in which each vehicle is modeled; and
a dataset calculation part configured to generate a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle to the model learning part so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving.
In the same field of endeavor Reddy teaches;
a model image in which each vehicle is modeled (Reddy, 7327, col. 2, “evaluated our approach on a large synthetic CAD dataset, showing similar performance benefits and improvements of up to 20% for occluded keypoints”; section 4.1, “We use 300 synthetic CAD models for training); and
a dataset calculation part configured to generate a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle to the model learning part so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving (Reddy, Fig.2, “The input is a ROI region from any detector, which is passed through multiple convolutional layers to predict the heatmaps with a confidence score. These confidences are passed through a graph encode-decoder network and trained using multi-view trifocal tensor loss for localization of occluded 2D keypoints”; Reddy, page 7328, Section 3. Occlusion-net, “Occlusion-Net consists of three main stages – visible keypoints detection, occluded 2D keypoint localization and 3D keypoint localization networks - as shown in Figure 2”; Reddy, Page 7330, CarFusion dataset: “To model a wide range of real occlusions, we collect an extensive dataset captured simultaneously by multiple mobile cameras at 60fps at 5 crowded traffic intersections”).
Hu and Reddy are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Hu with the invention of Reddy to make the invention that uses a model image in which each vehicle is modeled to obtain the invisible keypoints and generate a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle to the model learning part so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving; doing so can improve performance by accurately identifying occluded keypoints (Reddy, Conclusions); thus, one of ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 2, Hu in view of Reddy teaches the vehicle location calculation apparatus of claim 1,
wherein the model learning part includes a correction part configured to correct the visible keypoint of the modeled vehicle to be similar to a visible keypoint of an image of an actual vehicle (Reddy, page 7326, col. 1, para 1, “We use two views where a keypoint is seen (and labeled by humans) and compute the trifocal tensor using camera matrices to predict its location in the view where the keypoint is occluded. We call this the Trifocal tensor loss, which is minimized to correct the 2D keypoint positions from the initial detector… our approach explicitly predicts occluded keypoints. The predicted 2D keypoints (both occluded and visible) are then used in a graph network to estimate the 3D object shape and the camera projection matrix”; Reddy, Page 7329, Section 4.2 Datasets, “We use the 472 cars sampled from shapenet [4] and 3D annotated by [26]. We select 12 keypoints from the annotated 36 keypoints and render them from different viewpoints”), and
wherein the model learning part is configured to perform learning to output an invisible keypoint of the modeled vehicle by receiving a value of the correction as input thereof (Reddy, section 4.2, “We use the PCK metric [47] to analyze both the 2D and the 3D occluded keypoint locations. According to the PCK metric, a keypoint is considered correct if it lies within the radius _L of the ground truth. Here L is defined as the maximum of length and width of the bounding box and 0 < _ < 1. To evaluate the 3D reconstruction, we project the reconstructed keypoints into their respective views and compute the 2D PCK error… the occluded points predicted by Occlusion-Net provide much more correspondences to improve multi-view reconstruction”).
Regarding Claim 3, Hu in view of Reddy teaches the vehicle location calculation apparatus of claim 1, wherein the dataset calculation part includes a keypoint learning part configured to learn a visible keypoint in an image of an actual vehicle (Reddy, Page 7329, Section 4.2, “We use the 472 cars sampled from shapenet [4] and 3D annotated by [26]. We select 12 keypoints from the annotated 36 keypoints and render them from different viewpoints”).
Regarding Claim 4, Hu in view of Reddy teaches the vehicle location calculation apparatus of claim 1, wherein the dataset calculation part further includes a keypoint detection part configured to detect the visible keypoint of the target vehicle based on data regarding the visible keypoint learned by the keypoint learning portion (Reddy, page 7328, col. 1, section 3. Occlusion-Net, “Occlusion-Net consists of three main stages – visible keypoints detection, occluded 2D keypoint localization and 3D keypoint localization networks - as shown in Figure 2”; page 7330, Carfusion Dataset : “cars detected in these images were annotated with 12 keypoints each. Each annotation contains the visible and occluded keypoint locations on the car. We do not use the occluded keypoints for training the Occlusion-Net”).
Regarding Claim 5, Hu in view of Reddy teaches the vehicle location calculation apparatus of claim 1, further including:
a spatial coordinate calculation part configured to determine spatial coordinates of the target vehicle, based on the dataset including the visible keypoint and the invisible keypoint of the target vehicle (Hu, [0028] “ key points of the target objects are manually marked on multiple 2D images and the AR SLAM engine maps each of the key points to the 3D world coordinate system”; Hu, [0039] “the image segmentation module 42 automatically fits a minimal 3D bounding box based on the critical points (visible or occluded), the base plane, and the axis line(s) in the 3D world coordinate, output by the AR SLAM engine 28 and the image/point processing engine 30”).
Regarding Claim 8, Hu in view of Reddy teaches the vehicle location calculation apparatus of claim 1, further including:
a data generation part configured to set the visible keypoint and the invisible keypoint in the model image (Reddy, Fig.2, page 7328, section 3. Occlusion-Net “Occlusion-Net consists of three main stages – visible keypoints detection, occluded 2D keypoint localization and 3D keypoint localization networks - as shown in Figure 2. The 2D-Keypoint Graph Neural Network deforms the graph nodes to infer the 2D image locations of the occluded keypoints”; page 7330, CarFusion Dataset “Each annotation contains the visible and occluded keypoint locations on the car”).
Regarding Claim 9, Hu in view of Reddy teaches the vehicle location calculation apparatus of claim 1, wherein the data generation part includes:
a setting part configured to set a plurality of keypoint locations in the model image (Reddy, Fig. 2, page 7330, CarFusion Dataset, “Each annotation contains the visible and occluded keypoint locations on the car”); and
a labelling part configured to place the modeled vehicle in a 3D synthetic world, project 3D keypoint coordinates of the modeled vehicle onto a plane, confirm the visible keypoint and the invisible keypoint of the modeled vehicle to perform labeling (Reddy, page 7332, Conclusion, “presented a novel graph based architecture to predict the 2D and 3D locations of occluded keypoints. Since supervision for 2D occluded keypoints is challenging, computed the error using labeled visible keypoints from different views…proposed a self-supervised network to lift the 3D structure of the keypoints from the 2D keypoints. We demonstrated our approach on synthetic CAD data as well as a large image set capturing vehicles”).
Regarding Claim 10, Hu teaches:
A vehicle location calculation method, comprising:
setting, by a controller, a visible keypoint and an invisible keypoint in a model image in which each vehicle is modeled ([0040] “FIG. 4A represents a 2D image 50 showing vehicle 52 in a front perspective view. The key points 54 on the vehicle image 52 are manually marked incrementally using the image feature extraction module 40...The AR SLAM engine 28 maps the key points 54 to 3D coordinate system 58. FIG. 4B represents an additional 2D image 60 showing the vehicle 52 in a rear perspective view”);
learning, by the controller, to output the invisible keypoint based on the visible keypoint (Hu, [0010] “the method may include estimating occluded points in at least one of the plurality of 2D images by defining lines starting from visible marked points”; [0030] “the augmented reality application 24 includes a main controller 26”; [0044] “image feature extraction module 40 estimates essential occluded key points by defining lines starting from known key points”; [0052] “The training module is configured to train different machine learning models using a labeled image dataset and optionally an unlabeled image dataset).
However, Hu fails to explicitly teach:
a model image in which each vehicle is modeled; and
generating, by the controller, a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving.
In the same field of endeavor Reddy teaches;
a model image in which each vehicle is modeled (Reddy, 7327, col. 2, “evaluated our approach on a large synthetic CAD dataset, showing similar performance benefits and improvements of up to 20% for occluded keypoints”; section 4.1, “We use 300 synthetic CAD models for training); and
generating, by the controller, a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving (Reddy, Fig.2, “The input is a ROI region from any detector, which is passed through multiple convolutional layers to predict the heatmaps with a confidence score. These confidences are passed through a graph encode-decoder network and trained using multi-view trifocal tensor loss for localization of occluded 2D keypoints”; Reddy, page 7328, Section 3. Occlusion-net, “Occlusion-Net consists of three main stages – visible keypoints detection, occluded 2D keypoint localization and 3D keypoint localization networks - as shown in Figure 2”; Reddy, Page 7330, CarFusion dataset: “To model a wide range of real occlusions, we collect an extensive dataset captured simultaneously by multiple mobile cameras at 60fps at 5 crowded traffic intersections”).
Hu and Reddy are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Hu with the invention of Reddy to make the invention that uses a model image in which each vehicle is modeled to obtain the invisible keypoints and generate a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle to the model learning part so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving; doing so can improve performance by accurately identifying occluded keypoints (Reddy, Conclusions); thus, one of ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 11, Hu in view of Reddy teaches the vehicle location calculation method of claim 10, wherein the setting includes:
setting a plurality of keypoint locations in the model image (Reddy, Abstract, “Occlusion-Net successfully localizes keypoints in a single view under a diverse set of occlusion settings. We validate our approach on synthetic CAD data”);
placing the modeled vehicle in a 3D synthetic world (Reddy, page 7329, section 4.1 Datasets, “use 300 synthetic CAD models for training, 72 for validation and 100 for testing. We project the 3D keypoint annotations of the CAD model with visibility”);
projecting 3D keypoint coordinates of the modeled vehicle onto a plane, and confirming the visible keypoint and the invisible keypoint of the modeled vehicle to perform labeling (Reddy, Fig. 2, page 7327, col. 1, para 1 “We use two views where a keypoint is seen (and labeled by humans) and compute the trifocal tensor using camera matrices to predict its location in the view where the keypoint is occluded… our approach explicitly predicts occluded keypoints. The predicted 2D keypoints (both occluded and visible) are then used in a graph network to estimate the 3D object shape and the camera projection matrix”; page 7329, col. 1 last para, “projects a set of 3D keypoints W onto the image coordinates”).
Regarding Claim 12, Hu in view of Reddy teaches the vehicle location calculation method of claim 10, wherein the learning includes:
correcting the visible keypoint of the modeled vehicle to be similar to a visible keypoint of an image of an actual vehicle (Reddy, page 7326, col. 1, para 1, “We use two views where a keypoint is seen (and labeled by humans) and compute the trifocal tensor using camera matrices to predict its location in the view where the keypoint is occluded. We call this the Trifocal tensor loss, which is minimized to correct the 2D keypoint positions from the initial detector… our approach explicitly predicts occluded keypoints. The predicted 2D keypoints (both occluded and visible) are then used in a graph network to estimate the 3D object shape and the camera projection matrix”; Reddy, Page 7329, Section 4.2 Datasets, “We use the 472 cars sampled from shapenet [4] and 3D annotated by [26]. We select 12 keypoints from the annotated 36 keypoints and render them from different viewpoints”), and
inputting a value of the correction to output the invisible keypoint of the modeled vehicle (Reddy, section 4.2, “We use the PCK metric [47] to analyze both the 2D and the 3D occluded keypoint locations. According to the PCK metric, a keypoint is considered correct if it lies within the radius _L of the ground truth. Here L is defined as the maximum of length and width of the bounding box and 0 < _ < 1. To evaluate the 3D reconstruction, we project the reconstructed keypoints into their respective views and compute the 2D PCK error… the occluded points predicted by Occlusion-Net provide much more correspondences to improve multi-view reconstruction”).
Regarding Claim 13, Hu in view of Reddy teaches the vehicle location calculation method of claim 10, wherein the generating of the dataset includes learning a visible keypoint in an image of an actual vehicle (Reddy, Page 7329, Section 4.2, “We use the 472 cars sampled from shapenet [4] and 3D annotated by [26]. We select 12 keypoints from the annotated 36 keypoints and render them from different viewpoints”).
Regarding Claim 14, Hu in view of Reddy teaches the vehicle location calculation method of claim 10, wherein the dataset calculation part further includes a keypoint detection part configured to detect the visible keypoint of the target vehicle based on data regarding the visible keypoint learned by the keypoint learning portion (Reddy, page 7328, col. 1, section 3. Occlusion-Net, “Occlusion-Net consists of three main stages – visible keypoints detection, occluded 2D keypoint localization and 3D keypoint localization networks - as shown in Figure 2”; page 7330, Carfusion Dataset: “cars detected in these images were annotated with 12 keypoints each. Each annotation contains the visible and occluded keypoint locations on the car. We do not use the occluded keypoints for training the Occlusion-Net”).
Regarding Claim 15, Hu in view of Reddy teaches the vehicle location calculation method of claim 10, further including:
determining spatial coordinates of the target vehicle, based on the dataset including the visible keypoint and the invisible keypoint of the target vehicle (Hu, [0028] “ key points of the target objects are manually marked on multiple 2D images and the AR SLAM engine maps each of the key points to the 3D world coordinate system”; Hu, [0039] “the image segmentation module 42 automatically fits a minimal 3D bounding box based on the critical points (visible or occluded), the base plane, and the axis line(s) in the 3D world coordinate, output by the AR SLAM engine 28 and the image/point processing engine 30”).
Regarding Claim 18, Hu in view of Reddy teaches the vehicle location calculation method of claim 10, wherein the controller includes:
a processor (Hu, Fig. 1 processor 14); and
a non-transitory storage medium on which a program for performing the vehicle location calculation method of claim 10 and for being executed by the processor is recorded (Hu, claim 8, “one or more non-transitory computer-readable storage media; program instructions, stored on the one or more non-transitory computer-readable storage media”).
Claims 6 – 7 and 16 – 17 are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (US 20220300738 A1; hereafter referred to as Hu) in view of Reddy et al. (Reddy, N. D., Vo, M., & Narasimhan, S. G. (2019). Occlusion-net: 2d/3d occluded keypoint localization using graph networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7326-7335); hereafter referred to as Reddy) further in view of Peng et al. (See Machine Translation for CN112036389A; hereafter referred to as Peng)
Regarding Claim 6, Hu in view of Reddy teaches the vehicle location calculation apparatus of claim 5, but fails to explicitly teach:
wherein the spatial coordinate calculation part includes a first calculation part configured to:
determine three-dimensional (3D) camera coordinate values of keypoints of front and rear wheels on first and second sides of the target vehicle, by use of two-dimensional (2D) image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera, a height of the keypoints of the front and rear wheels on the first and second sides of the target vehicle being set as 0, and determine an angle between an x-axis and a vector connecting a center point of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and a center point of keypoints of the first and second front wheels of the target vehicle.
In the same field of endeavor, Peng teaches:
determine three-dimensional (3D) camera coordinate values of keypoints of front and rear wheels on first and second sides of the target vehicle, by use of two-dimensional (2D) image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera, a height of the keypoints of the front and rear wheels on the first and second sides of the target vehicle being set as 0 (Peng, page 5, step S12, “he output information includes four binary values and four vehicle side edges respectively indicating whether the four vehicle side edges are visible The image coordinates of the intersection with the road surface. The six sides of the vehicle are left side, right side, front, back, ground and top”, Peng, page 6, S13 : “The corresponding relationship of the vehicle orientation corresponding to the four binary values is shown in the following table, where 1 means visible and 0 means invisible”, Peng, page 6, S14: “When the determined vehicle orientation is a composite orientation, the vehicle heading angle is calculated according to the image coordinates of the intersection of the visible vehicle side edge and the road surface, the actual length and width of the vehicle, and the internal parameter matrix and external parameter matrix of the monocular camera”), and
determine an angle between an x-axis and a vector connecting a center point of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and a center point of keypoints of the first and second front wheels of the target vehicle (Peng, page 6, S14: “The actual length and width of the vehicle can be specifically obtained according to the corresponding relationship between the preset vehicle type and the actual length and width of the vehicle… The vehicle heading angle specifically refers to the angle between the front direction of the vehicle applying the present invention and the front direction of the vehicle identified in the actual scene image. When the determined vehicle heading is a single heading, the vehicle heading angle can be determined directly”).
Hu, Reddy and Peng are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Hu in view of Reddy with the invention of Peng to make the invention that calculates the spatial coordinates of the vehicle by using of two-dimensional (2D) image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera; doing so can efficiently detect three-dimensional information of the vehicle by reducing cost and enhancing stability (Peng, abstract); thus, one of ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 7, Hu in view of Reddy further in view of Peng teaches the vehicle location calculation apparatus of claim 6, wherein the spatial coordinate calculation part further includes a second calculation part configured to:
determine unknown values including a distance between first and second bumpers of the target vehicle and the first and second front wheels of the target vehicle, a height of the first and second bumpers from ground, and location values of the first and second bumpers disposed between the first and second front wheels, by use of 3D world coordinate values of the keypoints of the first and second front wheels, 2D image coordinate values of the first and second bumpers, the inverse matrix of the intrinsic and extrinsic parameters of the camera, and the angle (Peng, page 4, para 1, “The second image coordinate determination subunit is used to obtain the image coordinates of the intersection of the visible vehicle side edge and the road surface according to the rectangular frame and the side edge marked on the training image. The edge of the rectangular frame is parallel to the edge of the training image, so The two wheels on the side and the side are tangent to the ground points of the two wheels… When there are two visible vehicle side edges, the image coordinates of the intersection of the two visible vehicle side edges and the road surface are determined, which are the image coordinates of the intersection point of the left vertical line of the rectangular frame and the side edge and the rectangular frame respectively. The image coordinates of the intersection of the right vertical line of and the side”; Peng, page 4, last para, “the vehicle heading angle is calculated according to the image coordinates of the intersection of the visible vehicle side edge and the road surface, the actual length and width of the vehicle, and the internal parameter matrix and external parameter matrix of the monocular camera”), and
determine 3D camera coordinate values of the first and second bumpers based on the determined unknown values and 3D camera coordinate values of the keypoints of the first and second front wheels (Peng, page 9, para 5, “The vehicle heading angle calculation unit 54 is used to calculate the internal parameter matrix and external parameter matrix of the monocular camera based on the image coordinates of the intersection of the visible vehicle side edge and the road surface, the actual length and width of the vehicle, and the internal parameter matrix and external parameter matrix of the monocular camera Get the heading angle of the vehicle”; Peng page 9, para 9, “The second image coordinate determination subunit is used to obtain the image coordinates of the intersection of the visible vehicle side edge and the road surface according to the rectangular frame and the side marked on the training image. The edge of the rectangular frame is parallel to the edge of the training image, and the side and side The two wheels are tangent to the ground point of the two wheels”).
Regarding Claim 16, Hu in view of Reddy teaches the vehicle location calculation method of claim 15, but fails to explicitly teach:
wherein the determining of the spatial coordinates includes:
determining 3D camera coordinate values of keypoints of front and rear wheels on first and second sides of the target vehicle, by use of 2D image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera, a height of the keypoints of the front and rear wheels on the first and second sides of the target vehicle being set as 0, and
determining an angle between an x-axis and a vector connecting a center point of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and a center point of keypoints of the first and second front wheels of the target vehicle.
In the same field of endeavor, Peng teaches:
determining 3D camera coordinate values of keypoints of front and rear wheels on first and second sides of the target vehicle, by use of 2D image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera, a height of the keypoints of the front and rear wheels on the first and second sides of the target vehicle being set as 0 (Peng, page 5, step S12, “he output information includes four binary values and four vehicle side edges respectively indicating whether the four vehicle side edges are visible The image coordinates of the intersection with the road surface. The six sides of the vehicle are left side, right side, front, back, ground and top”, Peng, page 6, S13 : “The corresponding relationship of the vehicle orientation corresponding to the four binary values is shown in the following table, where 1 means visible and 0 means invisible”, Peng, page 6, S14: “When the determined vehicle orientation is a composite orientation, the vehicle heading angle is calculated according to the image coordinates of the intersection of the visible vehicle side edge and the road surface, the actual length and width of the vehicle, and the internal parameter matrix and external parameter matrix of the monocular camera”), and
determining an angle between an x-axis and a vector connecting a center point of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and a center point of keypoints of the first and second front wheels of the target vehicle (Peng, page 6, S14: “The actual length and width of the vehicle can be specifically obtained according to the corresponding relationship between the preset vehicle type and the actual length and width of the vehicle… The vehicle heading angle specifically refers to the angle between the front direction of the vehicle applying the present invention and the front direction of the vehicle identified in the actual scene image. When the determined vehicle heading is a single heading, the vehicle heading angle can be determined directly”).
Hu, Reddy and Peng are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Hu in view of Reddy with the invention of Peng to make the invention that calculates the spatial coordinates of the vehicle by using of two-dimensional (2D) image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera; doing so can efficiently detect three-dimensional information of the vehicle by reducing cost and enhancing stability (Peng, abstract); thus, one of ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 17, Hu in view of Reddy further in view of Peng teaches the vehicle location calculation method of claim 16, wherein the determining of the spatial coordinates further includes:
determining unknown values including a distance between first and second bumpers of the target vehicle and the first and second front wheels of the target vehicle, a height of the first and second bumpers from ground, and location values of the first and second bumpers disposed between the first and second front wheels, by use of 3D world coordinate values of the keypoints of the first and second front wheels, 2D image coordinate values of the first and second bumpers, the inverse matrix of the intrinsic and extrinsic parameters of the camera, and (Peng, page 4, para 1, “The second image coordinate determination subunit is used to obtain the image coordinates of the intersection of the visible vehicle side edge and the road surface according to the rectangular frame and the side edge marked on the training image. The edge of the rectangular frame is parallel to the edge of the training image, so The two wheels on the side and the side are tangent to the ground points of the two wheels… When there are two visible vehicle side edges, the image coordinates of the intersection of the two visible vehicle side edges and the road surface are determined, which are the image coordinates of the intersection point of the left vertical line of the rectangular frame and the side edge and the rectangular frame respectively. The image coordinates of the intersection of the right vertical line of and the side”; Peng, page 4, last para, “the vehicle heading angle is calculated according to the image coordinates of the intersection of the visible vehicle side edge and the road surface, the actual length and width of the vehicle, and the internal parameter matrix and external parameter matrix of the monocular camera”), and the angle, and
determining 3D camera coordinate values of the first and second bumpers based on the determined unknown values and 3D camera coordinate values of the keypoints of the first and second front wheels (Peng, page 9, para 5, “The vehicle heading angle calculation unit 54 is used to calculate the internal parameter matrix and external parameter matrix of the monocular camera based on the image coordinates of the intersection of the visible vehicle side edge and the road surface, the actual length and width of the vehicle, and the internal parameter matrix and external parameter matrix of the monocular camera Get the heading angle of the vehicle”; Peng page 9, para 9, “The second image coordinate determination subunit is used to obtain the image coordinates of the intersection of the visible vehicle side edge and the road surface according to the rectangular frame and the side marked on the training image. The edge of the rectangular frame is parallel to the edge of the training image, and the side and side The two wheels are tangent to the ground point of the two wheels”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220139061 A1 MODEL TRAINING METHOD AND APPARATUS, KEYPOINT POSITIONING METHOD AND APPARATUS, DEVICE AND MEDIUM
US 20220301304 A1 KEYPOINT-BASED SAMPLING FOR POSE ESTIMATION
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mehmood Jennifer can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664