DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 18 – 21 are objected to because of the following informalities:
Claim 18, add “and” at the end of seventh limitation after “…fourth residual network;”
Claim 19, replace “;” with “:” at the end of second limitation.
Claim 20, replace “:” with “;”at the end of third limitation.
Claim 21, replace “,” with “;” at the end of first limitation.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 – 21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 2, 7 – 9, 13, 15, 17 and 20 – 21, recites the limitation “the bounding box” in various limitations. There is lack of antecedence basis for this limitation in the claims as there is no prior definition for “bounding box”. It is unclear and confusing as claim 1 defines “a three-dimensional bounding box”. For the purpose of examination, the Examiner is reading all instances of “the bounding box” as “the three-dimensional bounding box”.
Claims 3 – 6, 10 – 12, 16 and 18 – 19 are rejected for being dependent on rejected base claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 - 21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process (concept performed in a human mind, including as observation, evaluation, judgment, opinion, organizing human activity and mathematical concepts and calculations). The claim(s) recite(s) a method, detect a focus of attention. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory).
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that claims 1 and 10 are directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the four statutory categories? YES. Claims 1, and 15 are directed to a method and device.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? YES, the claims recite steps that fall into the abstract idea category of mental processes.
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
The method in claim 1 (and device of claim 15) comprise a mental process that can be practicably performed in the human mind (or generic computers or components configured to perform the method) and, therefore, an abstract idea.
Regarding Claim 1: A method for detecting object poses, comprising:
acquiring image data, wherein the image data comprises a target object (insignificant extra solution activity of data gathering);
detecting two-dimensional first pose information of a three-dimensional bounding box in response to being projected onto the image data by inputting the image data into a two-dimensional detection model, wherein the bounding box is configured to detect the target object (mental process including observation and evaluation, and can be done mentally in the human mind or a generic computer program or components configured to perform the method; detecting two-dimensional first pose information …a two-dimensions detection model can be a generic computer program);
mapping the two-dimensional first pose information to three-dimensional second pose information (mental process including observation and evaluation, and can be done mentally in the human mind or a generic computer program or components configured to perform the method; mapping the two-dimensional pose…);
detecting third pose information of the target object based on the three-dimensional second pose information (mental process including observation and evaluation, and can be done mentally in the human mind or a generic computer program or components configured to perform the method; detecting pose information…); and
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same). As such, a person could mentally analyze an image and determine a fill level, either mentally or using a pen and paper. The mere nominal recitation that the various steps are being executed by a device/in a device (e.g. processing unit) does not take the limitations out of the mental process grouping.
The claimed functions –receiving data, detecting, mapping– could be performed conceptually by a human using pen and paper, and thus fall under abstract mental steps.
Conclusions: Thus, the claims are directed to an abstract idea.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Claims 1 and 15 does/do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application.
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception.
There is no indication that the method improves the functioning of a computer or classification itself.
Conclusion: Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? NO, the claims do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Claims 1 and 15 does/do not recite any additional elements that are not well-understood, routine or conventional.
The claims lack an inventive concept sufficient to transform the abstract idea into patent-eligible subject matter.
The claims are functionally generic with no details about architecture, dataset specifics, or a novel arrangement of components.
Conclusion: The claims does not add significantly more than the abstract idea.
Final Determination: INELIGIBLE under 35 U.S.C. 101. The Claims 1 and 15 are: directed toward an abstract idea (mental process and data manipulation) using conventional tool (model) in a generic way, without integration into a practical application or an inventive concept.
Regarding Claims 11 – 13, 16 – 17: the additional elements recited in the claims do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations merely recite that the functions are performed by a processing unit and does not demonstrate a technological improvement. The claims are functionally generic with no details about architecture, dataset specifics, or a novel arrangement of components. Since the claims are directed toward an abstract idea (mental process and data manipulation) using conventional tools in a generic way, without integration into a practical application or an inventive concept, they are ineligible under 35 U.S.C. 101.
Claims 2 – 10, and 18 – 21: the additional elements recited in the claims add significantly more to the mental process and therefore are eligible under 35 U.S.C. 101. However, since these claims are dependent on rejected base claims, claims 2 – 8, 10 and 18 – 21 are objected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
(g)(1) during the course of an interference conducted under section 135 or section 291, another inventor involved therein establishes, to the extent permitted in section 104, that before such person’s invention thereof the invention was made by such other inventor and not abandoned, suppressed, or concealed, or (2) before such person’s invention thereof, the invention was made in this country by another inventor who had not abandoned, suppressed, or concealed it. In determining priority of invention under this subsection, there shall be considered not only the respective dates of conception and reduction to practice of the invention, but also the reasonable diligence of one who was first to conceive and last to reduce to practice, from a time prior to conception by the other.
A rejection on this statutory basis (35 U.S.C. 102(g) as in force on March 15, 2013) is appropriate in an application or patent that is examined under the first to file provisions of the AIA if it also contains or contained at any time (1) a claim to an invention having an effective filing date as defined in 35 U.S.C. 100(i) that is before March 16, 2013 or (2) a specific reference under 35 U.S.C. 120, 121, or 365(c) to any patent or application that contains or contained at any time such a claim.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 15 and 16 are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Rad et al. (US 20180137644 A1; hereafter referred to as Rad).
Regarding Claim 1, Rad teaches:
A method for detecting object poses, comprising:
acquiring image data, wherein the image data comprises a target object (Rad, [0010] “obtaining an image including the object”);
detecting two-dimensional first pose information of a three-dimensional bounding box in response to being projected onto the image data by inputting the image data into a two-dimensional detection model, wherein the bounding box is configured to detect the target object (Rad, [0010] “determining a plurality of two-dimensional projections of a three-dimensional bounding box of the object; The plurality of two-dimensional projections of the three-dimensional bounding box are determined by applying a trained regressor to the image. The trained regressor is trained to predict two-dimensional projections of the three-dimensional bounding box of the object in a plurality of poses”).
mapping the two-dimensional first pose information to three-dimensional second pose information (Rad, [0016] “applying the mappings to the obtained image to determine corresponding two-dimensional projections of the three-dimensional bounding box of the object in the obtained image, wherein the obtained image is mapped to the corresponding two-dimensional projections using the mappings”; see [0020] Rad, [0068] “The image-to-2D-projection mapping engine 108 can generate mappings 110 based on the training. The mappings 106 can map an image with an object to the corresponding two-dimensional projections of the points on the bounding box of the object (e.g., the corners). During run-time when input images are processed for determining 3D poses of a target object in the images, the 3D poses can be computed by mapping an input image with an object to the corresponding two-dimensional projections of the points on the bounding box”); and
detecting third pose information of the target object based on the three-dimensional second pose information (Rad, [0010] estimating the three-dimensional pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box”; Rad, [0077] “given an image window W centered on a target object of interest, the 3D pose of the target object can then be estimated for the correspondences between the 3D points M.sub.i and the predicted m.sub.i(f.sub.Θ(W)) using a pose estimation technique that estimates a pose given a set of 3D points in a world coordinate frame and their corresponding 2D points in the image; Rad, [0078] One example of such a pose estimation technique that estimates a pose given a set of 3D points and their corresponding 2D points in the image includes perspective-n-point (PnP) algorithm. The PnP algorithm estimates a pose of a calibrated camera (relative to a target object in a certain pose) using a given set of n 3D points of the object's bounding box in world coordinates and their corresponding 2D projections in the image”).
Regarding Claim 15, Rad teaches:
A computer device for detecting object poses, comprising:
at least one processor (Rad, [0011] “an apparatus is provided that includes a processor”);
a memory, configured to store at least one program therein (Rad, [0121] The computing device 1200 may further include (and/or be in communication with) one or more non-transitory storage devices 1225, which may comprise, without limitation, local and/or network accessible storage, and/or may include, without limitation, a disk drive, a drive array, an optical storage device, a solid-form storage device such as a random access memory (“RAM”) and/or a read-only memory (“ROM”), which may be programmable, flash-updateable and/or the like”);
wherein the at least one processor, when loading and running the at least one program, is caused to perform:
acquiring image data, wherein the image data comprises a target object (Rad, [0010] “obtaining an image including the object”);
detecting two-dimensional first pose information of a three-dimensional bounding box in response to being projected onto the image data by inputting the image data into a two-dimensional detection model, wherein the bounding box is configured to detect the target object (Rad, [0010] “determining a plurality of two-dimensional projections of a three-dimensional bounding box of the object; The plurality of two-dimensional projections of the three-dimensional bounding box are determined by applying a trained regressor to the image. The trained regressor is trained to predict two-dimensional projections of the three-dimensional bounding box of the object in a plurality of poses”).
mapping the two-dimensional first pose information to three-dimensional second pose information (Rad, [0016] “applying the mappings to the obtained image to determine corresponding two-dimensional projections of the three-dimensional bounding box of the object in the obtained image, wherein the obtained image is mapped to the corresponding two-dimensional projections using the mappings”; see [0020] Rad, [0068] “The image-to-2D-projection mapping engine 108 can generate mappings 110 based on the training. The mappings 106 can map an image with an object to the corresponding two-dimensional projections of the points on the bounding box of the object (e.g., the corners). During run-time when input images are processed for determining 3D poses of a target object in the images, the 3D poses can be computed by mapping an input image with an object to the corresponding two-dimensional projections of the points on the bounding box”); and
detecting third pose information of the target object based on the three-dimensional second pose information (Rad, [0010] estimating the three-dimensional pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box”; Rad, [0077] “given an image window W centered on a target object of interest, the 3D pose of the target object can then be estimated for the correspondences between the 3D points M.sub.i and the predicted m.sub.i(f.sub.Θ(W)) using a pose estimation technique that estimates a pose given a set of 3D points in a world coordinate frame and their corresponding 2D points in the image; Rad, [0078] One example of such a pose estimation technique that estimates a pose given a set of 3D points and their corresponding 2D points in the image includes perspective-n-point (PnP) algorithm. The PnP algorithm estimates a pose of a calibrated camera (relative to a target object in a certain pose) using a given set of n 3D points of the object's bounding box in world coordinates and their corresponding 2D projections in the image”).
Regarding Claim 16, Rad teaches:
A non-transitory computer-readable storage medium, storing at least one computer program therein, wherein the at least one computer program (Rad, [0122] “the computing device 1200 will further comprise a non-transitory working memory 1235, which may include a RAM or ROM device”), when loaded and run by a processor (Rad, [0123] “The computing device 1200 may comprise software elements, shown as being currently located within the working memory 1235, including an operating system 1240, device drivers, executable libraries, and/or other code, such as one or more application programs 1245, which may comprise computer programs provided by various embodiments”), causes the processor to perform the method for detecting object poses as defined in claim 1 (Refer to rejection for claim 1 above).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 2, 8, 9, 17 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Rad et al. (US 20180137644 A1; hereafter referred to as Rad) in view of Chen et al. (See Machine Translation for CN 111968235 A; hereafter referred to as Chen).
Regarding Claim 2, Rad teaches the method according to claim 1, but fails to explicitly teach:
the two-dimensional detection model comprises an encoder, a decoder, and a prediction network;
detecting the two-dimensional first pose information of the bounding box by inputting the image data into the two-dimensional detection model comprises:
acquiring a first image feature by encoding the image data in the encoder;
acquiring a second image feature by decoding the first image feature in the decoder; and
mapping, in the prediction network, the second image feature to the two-dimensional first pose information of the bounding box.
In the same field of endeavor Chen teaches:
the two-dimensional detection model comprises an encoder, a decoder, and a prediction network (Chen, page 7, para 2, “Two-dimensional image, the mapping relationship between two-dimensional image to three-dimensional coordinate map is fitted through the codec network, and the two-dimensional image is processed through the mapping relationship to obtain the predicted three-dimensional coordinate map corresponding to the two-dimensional image, and the predicted three-dimensional map is processed by the PnP algorithm To determine the pose corresponding to the two-dimensional image”; Chen, page 7, para 3, “the encoding and decoding network is also the encode-decode network, through the encode-decode network to fit the transformation of the RGB image to the three-dimensional coordinate map, where the encoding part, which is the encode part, can use the residual network (Residual Network, referred to as Resnet) or high-resolution network (High-ResoultionNet, referred to as HRnet), the decoding part, which is the decoded part, can be up-sampling and convolution, and feature pyramid networks”); and
detecting the two-dimensional first pose information of the bounding box by inputting the image data into the two-dimensional detection model comprises:
acquiring a first image feature by encoding the image data in the encoder (Chen, page 7, para 3, “the encoding part, which is the encode part, can use the residual network (Residual Network, referred to as Resnet);
acquiring a second image feature by decoding the first image feature in the decoder (Chen, page 7, para 3, “the decoding part, which is the decoded part, can be up-sampling and convolution, and feature pyramid networks ((FeaturePyramidNetworks, FPN) can also be added Use multi-scale information”); and
mapping, in the prediction network, the second image feature to the two-dimensional first pose information of the bounding box (Chen, page 7, para 2, “a neural network model based on the object detection algorithm, the mapping relationship and the PnP algorithm is established, wherein the object detection algorithm is used to determine the detection frame of the training object in the two-dimensional image, and the detection frame is cut out to generate the two for training the mapping relationship”; Chen, page 9, para 5, “The posture generation module 51 is used to input a two-dimensional image into the neural network model to obtain the posture of the target object. The two-dimensional image of the training object is used to train the neural network model, and the two-dimensional image and the depth image of the training object are used to determine the training object The three-dimensional model of, and preset the three-dimensional bounding box according to the actual size of the training object, and mark the three-dimensional model according to the three-dimensional bounding box”).
Rad and Chen are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rad with the invention of Chen to make the invention uses the two dimensional detection model with an encode, decoder and a prediction network to determine the image features and mapping the second image feature to the two-dimensional first pose information of the bounding box; doing so can yield predictable results of accurately estimating the pose of a target object from an input image (Chen Abstract); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 8, Rad teaches the method according to claim 1, but fails to explicitly teach: wherein
the two-dimensional detection model comprises a target detection model and an encoding model, wherein the target detection model and the encoding model are cascaded; and
detecting the two-dimensional first pose information of the bounding box by inputting the image data into the two-dimensional detection model comprises:
detecting, in the target detection model, a part of the two-dimensional first pose information of the bounding box in the image data and a region in which the target object is located in the image data;
extracting data in the region in the image data as region data; and
acquiring the part of the two-dimensional first pose information of the bounding box by encoding the region data in the encoding model.
In the same field of endeavor, Chen teaches:
the two-dimensional detection model comprises a target detection model and an encoding model, wherein the target detection model and the encoding model are cascaded (Chen, page 7, para 2, “a neural network model based on the object detection algorithm, the mapping relationship and the PnP algorithm is established, wherein the object detection algorithm is used to determine the detection frame of the training object in the two-dimensional image, and the detection frame is cut out to generate the two for training the mapping relationship. Two-dimensional image, the mapping relationship between two-dimensional image to three-dimensional coordinate map is fitted through the codec network”); and
detecting the two-dimensional first pose information of the bounding box by inputting the image data into the two-dimensional detection model comprises:
detecting, in the target detection model, a part of the two-dimensional first pose information of the bounding box in the image data and a region in which the target object is located in the image data (Chen, page 3, para 15, “The posture generation module is configured to input the two-dimensional image into a neural network model to obtain the posture of the target object, wherein the neural network model is trained using a two-dimensional image of the training object, and the two-dimensional image is used Determine the three-dimensional model of the training object with the depth image of the training object, preset a three-dimensional bounding box according to the actual size of the training object, and label the three-dimensional model according to the three dimensional bounding box”);
extracting data in the region in the image data as region data (Chen, page 6, para 4, “the two-dimensional image is input to the neural network model to obtain 6D posture information of the target object, and the training data of the neural network model is based on a large number of two-dimensional images and depth images obtained by the depth camera, two-dimensional images and depth The image is reconstructed from the three-dimensional model of the training object, and the three dimensional bounding box is preset according to the actual size of the training object, and the three-dimensional image of the training object is annotated according to the three-dimensional bounding box, and then a large amount of real training data can be obtained, avoiding the related technology The fact that there is less real data with annotations solves the problem of using a full convolutional network to extract the heat map of the eight vertices of the three-dimensional bounding box of each target object on the two-dimensional image”); and
acquiring the part of the two-dimensional first pose information of the bounding box by encoding the region data in the encoding model (Chen, page 6, para 4, “and the three dimensional bounding box is preset according to the actual size of the training object, and the three-dimensional image of the training object is annotated according to the three-dimensional bounding box, and then a large amount of real training data can be obtained, avoiding the related technology The fact that there is less real data with annotations solves the problem of using a full convolutional network to extract the heat map of the eight vertices of the three-dimensional bounding box of each target object on the two-dimensional image”).
Rad and Chen are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rad with the invention of Chen to make the invention uses the two dimensional detection model for detecting the two-dimensional first pose information of the bounding box, extracting data in the region in the image data as region data, and acquiring the part of the two-dimensional first pose information of the bounding box by encoding the region data in the encoding model; doing so can yield predictable results of accurately estimating the pose of a target object from an input image (Chen Abstract); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 9, Rad in view of Chen teaches the method according to claim 8, wherein
detecting, in the target detection model, the part of the two-dimensional first pose information of the bounding box in the image data and the region in which the target object is located in the image data comprises:
detecting, in the target detection model, a depth and a scale of the bounding box in the image data (Rad, [0056] “The prediction of this forest is integrated in an energy function together with a term that compares the depth map with a rendering of the object and a term that penalizes pixels that lie on the object rendering but predicted by the forest to not be an object point”; Rad, [0059] “predicting the 2D projections of the bounding box of an object (e.g., the corners of the bounding box, or other suitable points of the bounding box), the systems and methods described herein avoid the need for a scale factor to balance the position and orientation errors, and the pose is more accurate when predicted in this form”); and
acquiring the part of the two-dimensional first pose information of the bounding box by encoding the region data in the encoding model comprises:
acquiring a center point and a vertex of the bounding box by encoding the region data in the encoding model (Rad, [0071] “Various implementations can be used to estimate the 2D projections of a target object's 3D bounding box given an image or an image window centered on the target object, and to estimate the object pose using the 2D projections”; Rad, [0098] “The last fully connected layer can output 16 values in the examples in which the pose engine 332 predicts 2D projections of 8 vertices of the corners of an object's 3D bounding box, or their 2D updates”).
Regarding Claim 17, Rad teaches the computer device for detecting object poses according to claim 15, but fails to explicitly teach:
the two-dimensional detection model comprises an encoder, a decoder, and a prediction network;
the at least one processor, when loading and running the at least one program, is caused to perform:
acquiring a first image feature by encoding the image data in the encoder;
acquiring a second image feature by decoding the first image feature in the decoder; and
mapping, in the prediction network, the second image feature to the two-dimensional first pose information of the bounding box.
In the same field of endeavor Chen teaches:
the two-dimensional detection model comprises an encoder, a decoder, and a prediction network (Chen, page 7, para 2, “Two-dimensional image, the mapping relationship between two-dimensional image to three-dimensional coordinate map is fitted through the codec network, and the two-dimensional image is processed through the mapping relationship to obtain the predicted three-dimensional coordinate map corresponding to the two-dimensional image, and the predicted three-dimensional map is processed by the PnP algorithm To determine the pose corresponding to the two-dimensional image”; Chen, page 7, para 3, “the encoding and decoding network is also the encode-decode network, through the encode-decode network to fit the transformation of the RGB image to the three-dimensional coordinate map, where the encoding part, which is the encode part, can use the residual network (Residual Network, referred to as Resnet) or high-resolution network (High-ResoultionNet, referred to as HRnet), the decoding part, which is the decoded part, can be up-sampling and convolution, and feature pyramid networks”); and
the at least one processor, when loading and running the at least one program, is caused to perform:
acquiring a first image feature by encoding the image data in the encoder (Chen, page 7, para 3, “the encoding part, which is the encode part, can use the residual network (Residual Network, referred to as Resnet);
acquiring a second image feature by decoding the first image feature in the decoder (Chen, page 7, para 3, “the decoding part, which is the decoded part, can be up-sampling and convolution, and feature pyramid networks ((FeaturePyramidNetworks, FPN) can also be added Use multi-scale information”); and
mapping, in the prediction network, the second image feature to the two-dimensional first pose information of the bounding box (Chen, page 7, para 2, “a neural network model based on the object detection algorithm, the mapping relationship and the PnP algorithm is established, wherein the object detection algorithm is used to determine the detection frame of the training object in the two-dimensional image, and the detection frame is cut out to generate the two for training the mapping relationship”; Chen, page 9, para 5, “The posture generation module 51 is used to input a two-dimensional image into the neural network model to obtain the posture of the target object. The two-dimensional image of the training object is used to train the neural network model, and the two-dimensional image and the depth image of the training object are used to determine the training object The three-dimensional model of, and preset the three-dimensional bounding box according to the actual size of the training object, and mark the three-dimensional model according to the three-dimensional bounding box”).
Rad and Chen are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rad with the invention of Chen to make the invention uses the two dimensional detection model with an encode, decoder and a prediction network to determine the image features and mapping the second image feature to the two-dimensional first pose information of the bounding box; doing so can yield predictable results of accurately estimating the pose of a target object from an input image (Chen Abstract); thus one of the ordinary skill in the art would have been motivated to combine the references.
Regarding Claim 21, Rad teaches the computer device for detecting object poses according to claim 15, but fails to explicitly teach: wherein
the two-dimensional detection model comprises a target detection model and an encoding model, wherein the target detection model and the encoding model are cascaded; and
the at least one processor, when loading and running the at least one program, is caused to perform:
detecting, in the target detection model, a part of the two-dimensional first pose information of the bounding box in the image data and a region in which the target object is located in the image data;
extracting data in the region in the image data as region data; and
acquiring the part of the two-dimensional first pose information of the bounding box by encoding the region data in the encoding model.
In the same field of endeavor, Chen teaches:
the two-dimensional detection model comprises a target detection model and an encoding model, wherein the target detection model and the encoding model are cascaded (Chen, page 7, para 2, “a neural network model based on the object detection algorithm, the mapping relationship and the PnP algorithm is established, wherein the object detection algorithm is used to determine the detection frame of the training object in the two-dimensional image, and the detection frame is cut out to generate the two for training the mapping relationship. Two-dimensional image, the mapping relationship between two-dimensional image to three-dimensional coordinate map is fitted through the codec network”); and
the at least one processor, when loading and running the at least one program, is caused to perform:
detecting, in the target detection model, a part of the two-dimensional first pose information of the bounding box in the image data and a region in which the target object is located in the image data (Chen, page 3, para 15, “The posture generation module is configured to input the two-dimensional image into a neural network model to obtain the posture of the target object, wherein the neural network model is trained using a two-dimensional image of the training object, and the two-dimensional image is used Determine the three-dimensional model of the training object with the depth image of the training object, preset a three-dimensional bounding box according to the actual size of the training object, and label the three-dimensional model according to the three dimensional bounding box”);
extracting data in the region in the image data as region data (Chen, page 6, para 4, “the two-dimensional image is input to the neural network model to obtain 6D posture information of the target object, and the training data of the neural network model is based on a large number of two-dimensional images and depth images obtained by the depth camera, two-dimensional images and depth The image is reconstructed from the three-dimensional model of the training object, and the three dimensional bounding box is preset according to the actual size of the training object, and the three-dimensional image of the training object is annotated according to the three-dimensional bounding box, and then a large amount of real training data can be obtained, avoiding the related technology The fact that there is less real data with annotations solves the problem of using a full convolutional network to extract the heat map of the eight vertices of the three-dimensional bounding box of each target object on the two-dimensional image”); and
acquiring the part of the two-dimensional first pose information of the bounding box by encoding the region data in the encoding model (Chen, page 6, para 4, “and the three dimensional bounding box is preset according to the actual size of the training object, and the three-dimensional image of the training object is annotated according to the three-dimensional bounding box, and then a large amount of real training data can be obtained, avoiding the related technology The fact that there is less real data with annotations solves the problem of using a full convolutional network to extract the heat map of the eight vertices of the three-dimensional bounding box of each target object on the two-dimensional image”).
Rad and Chen are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rad with the invention of Chen to make the invention uses the two dimensional detection model for detecting the two-dimensional first pose information of the bounding box, extracting data in the region in the image data as region data, and acquiring the part of the two-dimensional first pose information of the bounding box by encoding the region data in the encoding model; doing so can yield predictable results of accurately estimating the pose of a target object from an input image (Chen Abstract); thus one of the ordinary skill in the art would have been motivated to combine the references.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Rad et al. (US 20180137644 A1; hereafter referred to as Rad) in view of Francesc et al. (Moreno-Noguer, Francesc, Vincent Lepetit, and Pascal Fua. "Accurate Non-Iterative 𝑂 (𝑛) Solution to the P𝑛P Problem." 2007 IEEE 11th International Conference on Computer Vision. Ieee, 2007; hereafter referred to as Francesc).
Regarding Claim 10, Rad teaches the method according to claim 1, wherein
the first pose information comprises a center point, a vertex, and a depth (Rad, [0071] Various implementations can be used to estimate the 2D projections of a target object's 3D bounding box given an image or an image window centered on the target object, and to estimate the object pose using the 2D projections”; Rad, [0098] “The last fully connected layer can output 16 values in the examples in which the pose engine 332 predicts 2D projections of 8 vertices of the corners of an object's 3D bounding box, or their 2D updates. The rectified linear unit (ReLU) activation function can be used”; Rad, [0057] “designed for RGB-D data that includes depth information”); and
mapping the two-dimensional first pose information to the three-dimensional second pose information comprises:
a world coordinate system and a camera coordinate system, separately (Rad, [0078] “The PnP algorithm estimates a pose of a calibrated camera (relative to a target object in a certain pose) using a given set of n 3D points of the object's bounding box in world coordinates and their corresponding 2D projections in the image. The calibrated intrinsic camera parameters can also be used. The camera pose includes six degrees-of-freedom, including the rotation (e.g., roll, pitch, and yaw) and the 3D translation of the camera with respect to the world. Any suitable PnP technique can be used, including P3P, efficient PnP (EPnP), or other suitable PnP technique”);
However, Rad fails to explicitly teach:
querying control points under a world coordinate system and a camera coordinate system, separately;
representing the center point and the vertex as a weighted sum of the control points under the world coordinate system and the camera coordinate system;
constructing a constraint relationship of the depth, the center point and the vertex between the world coordinate system and the camera coordinate system;
acquiring a linear equation by connecting the constraint relationships in series; and
mapping the vertex to 3D space by solving the linear equation.
In the same field of endeavor, Francesc teaches:
querying control points under a world coordinate system and a camera coordinate system, separately (Francesc, section 3.0 col. 2, para 1, “The solution of our problem can be expressed as a vector that lies in the kernel of a matrix of size 2n × 12 or 2n×9. We denote this matrix as M and can be easily computed from the 3D world coordinates of the reference points and their 2D image projections. More precisely, it is a weighted sum of the null eigenvectors of M. Given that the correct linear combination is the one that yields 3D camera coordinates for the control points that preserve their distances, we can find the appropriate weights by solving small systems of quadratic equations”);
representing the center point and the vertex as a weighted sum of the control points under the world coordinate system and the camera coordinate system, separately (Francesc, section 3.1 “When necessary, we will specify that the point coordinates are expressed in the world coordinate system by using the w superscript, and in the camera coordinate system by using the c superscript. We express each reference point as a weighted sum of the control points”);
constructing a constraint relationship of the depth, the center point and the vertex between the world coordinate system and the camera coordinate system (Francesc, section 3.1 and 3.2);
acquiring a linear equation by connecting the constraint relationships in series (Francesc, section 3.3); and
mapping the vertex to 3D space by solving the linear equation (Francesc, section 3.1 and 3.2).
Rad and Francesc are considered analogous art as they are reasonably pertinent to the same field of endeavor of image processing. Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Rad with the invention of Francesc to make the invention uses the control points under a world coordinate system and a camera coordinate system, representing the center point and the vertex as a weighted sum of the control points under the world coordinate system and the camera coordinate system; constructing a constraint relationship of the depth, the center point and the vertex between the world coordinate system and the camera coordinate system; acquiring a linear equation by connecting the constraint relationships in series; and mapping the vertex to 3D space by solving the linear equation; doing so can yield predictable results of accurately estimating the pose of a target object (Francesc Section 5); thus one of the ordinary skill in the art would have been motivated to combine the references.
Allowable Subject Matter
Claims 3 – 7, 11 – 13, and 18 – 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and by overcoming any claim objections and claim rejections under 35 U.S.C. 112(b).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20200311977 A1 DETECTING POSE OF 3D OBJECTS USING A GEOMETRY IMAGE
US 20200410273 A1 TARGET DETECTION METHOD AND APPARATUS, COMPUTER-READABLE STORAGE MEDIUM, AND COMPUTER DEVICE
US 20190355150 A1 DETECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAISALI RAO KOPPOLU whose telephone number is (571)270-0273. The examiner can normally be reached Monday - Friday 8:30 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
VAISALI RAO. KOPPOLU
Examiner
Art Unit 2664
/VAISALI RAO KOPPOLU/Examiner of Art Unit 2664
/XIAO LIU/Primary Examiner, Art Unit 2664
Prosecution Insights