DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Marullo et al. (“6D object position estimation from 2D images: a literature review”) in view of Konishi et al. (“Fast 6D Pose Estimation from a Monocular Image using Hierarchical Pose Trees”).
Regarding Claim 1, Marullo teaches A photographing position and posture estimation (“6D object position estimation from 2D images” Marullo Title. “The problem itself is simple and consists of determining the 3D rotation and translation of an object which shape is known in relation to the camera, using details observable from the reference 2D image.” Marullo 1 Introduction. Marullo Figs. 2-5 show the 2D images could be photographs.) apparatus comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to (“The 6D pose estimation of an object from an image is a central problem in many domains of Computer Vision (CV) and researchers have struggled with this issue for several years.” Marullo Abstract.):
calculate shape feature values from specific 3D (three-dimensional) data (
PNG
media_image1.png
746
1148
media_image1.png
Greyscale
“Methods in this category take advantage of local features (keypoints, grey values, edges, or intersections of straight lines) extracted from the regions of interest or all pixels in the image, and then compared with the features found on a 3D model of the object to establish 2D-3D matches [16, 20, 66].” Marullo p. 24628.);
calculate a degree of similarity between the virtual image information and an input image (
Marullo teaches calculating a degree of similarity to identify 2D-3D matches, stating “The input image is first passed through a feature extraction step; then the extracted features are compared with those annotated on a 3D model of the object to find 2D-3D matches.” Marullo Fig. 2.).
Marullo does not explicitly disclose, but Konishi teaches 2D-3D matching to generate virtual image information by associating the shape feature values projected (rendered) onto an image area (as shown in Konishi Fig. 2 (b)) when the 3D data is converted into an image based on specific virtual camera information (viewpoints) with respective pixel positions in the image area (pixel positions in the rendered images) (Konishi Figs. 2, 4
PNG
media_image2.png
398
738
media_image2.png
Greyscale
PNG
media_image3.png
532
738
media_image3.png
Greyscale
The specific virtual camera information is mapped to specific viewpoints used for rendering the 3D object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Konishi’s 3D-2D matching algorithm with Marullo. One of ordinary skill in the art would be motivated to quickly and accurately pose estimate 6D pose. Konishi 1 Introduction. In addition, Marullo specifically cites Konishi as an example for determining 2D-3D matches.
Regarding Claim 2, Marullo in view of Konishi teaches The photographing position and posture estimation apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to
convert the shape feature values (e.g., outline and/or hole as shown in Konishi Fig. 2) based on a photographing position and a photographing posture (randomized viewpoints) included in the virtual camera information, and generate the virtual image information (as shown in Fig. 2 (b)) by using the converted shape feature values (e.g., converted/modified outline and/or hole as shown in Konishi Fig. 2 (b)) ( Konishi Figs. 2, 4. The virtual camera information is mapped to a viewpoints used for rendering the 3D object.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Konishi’s 3D-2D matching algorithm with Marullo. One of ordinary skill in the art would be motivated to quickly and accurately pose estimate 6D pose. Konishi 1 Introduction. In addition, Marullo specifically cites Konishi as an example for determining 2D-3D matches.
Regarding Claim 3, Marullo in view of Konishi teaches The photographing position and posture estimation apparatus according to claim 1, wherein at least one processor is further configured to execute the instructions to calculate the shape feature values at respective data points so as to express a distribution of other data points (data points based on a distribution of viewpoints) around a specific data point of the 3D data (around center of the 3D object as shown Fig. 2) (
Marullo:
PNG
media_image4.png
250
1148
media_image4.png
Greyscale
Konishi:
PNG
media_image2.png
398
738
media_image2.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Konishi’s 3D-2D matching algorithm with Marullo. One of ordinary skill in the art would be motivated to quickly and accurately pose estimate 6D pose. Konishi 1 Introduction. In addition, Marullo specifically cites Konishi as an example for determining 2D-3D matches.
Regarding Claim 4, Marullo in view of Konishi teaches The photographing position and posture estimation apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to:
generate a plurality of pieces of virtual camera information (different viewpoints as shown in Marullo fig. 3; Konishi Figs. 2, 4.) in such a manner that a data point of which a shape is more characteristic (certain viewpoint shows more characteristic of an aspect of the 3D object model) than those of other data points among the data points of the 3D data is included in a photographing range (for example, an overhead camera view shows more details/characteristics of the top surface of an object than from the bottom viewpoint);
generate a plurality of pieces of virtual image (viewpoint images) information corresponding to respective pieces of virtual camera information based on the respective pieces of virtual camera information (different viewpoints) (Marullo fig. 3; Konishi Figs. 2, 4.); and
calculate a degree of similarity between each of the plurality of pieces of virtual image information and the input image (Marullo fig. 3 discloses finding the “best match”.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Konishi’s 3D-2D matching algorithm with Marullo. One of ordinary skill in the art would be motivated to quickly and accurately pose estimate 6D pose. Konishi 1 Introduction. In addition, Marullo specifically cites Konishi as an example for determining 2D-3D matches.
Regarding Claim 5, Marullo in view of Konishi teaches The photographing position and posture estimation apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to
generate the virtual image information for an area corresponding to a common photographing range (photographing range of the 3D object, e.g., chair) of first virtual camera information (with respect to first viewpoint) and second virtual camera information (with respect to second viewpoint) when the 3D data is converted (rendered) into an image ( Marullo Fig. 3; Konishi Figs. 2, 4).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Konishi’s 3D-2D matching algorithm with Marullo. One of ordinary skill in the art would be motivated to quickly and accurately pose estimate 6D pose. Konishi 1 Introduction. In addition, Marullo specifically cites Konishi as an example for determining 2D-3D matches.
Regarding Claim 7, Marullo in view of Konishi teaches The photographing position and posture estimation apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to
display a virtual image (Konishi Fig. 3 (c)) in which the shape feature values associated with the respective pixels of the virtual image information are converted into color information (Konishi fig. 3(c)) through dimensional compression (reduced to 2D image; also with reduced color dimension compared to fig. 3(a)) (
Konishi Fig. 3:
PNG
media_image5.png
642
924
media_image5.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Konishi’s 3D-2D matching algorithm with Marullo. One of ordinary skill in the art would be motivated to quickly and accurately pose estimate 6D pose. Konishi 1 Introduction. In addition, Marullo specifically cites Konishi as an example for determining 2D-3D matches.
Claim 9 is substantially similar to Claim 1. The rejection analyses based on Marullo in view of Konishi for Claim 1 are applied to Claim 9. In addition, Claim 9 recites “A photographing position and posture estimation method . . .,” where Marullo and Konishi teach the method to estimate 6D pose of an object in a 2D image. Marullo Abstract; Konishi Abstract.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Marullo in view of Konishi as applied to Claim 1, in further view of Wang et al. (“GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation”) and Hu et al. (“Wide-Depth-Range 6D Object Pose Estimation in Space”).
Regarding Claim 6, Marullo in view of Konishi teaches The photographing position and posture estimation apparatus according to claim 1.
Marullo in view of Konishi does not explicitly disclose, but Wang teaches wherein the at least one processor is further configured to execute the instructions to:
select, of two input images (Wang Fig. 2:
PNG
media_image6.png
160
84
media_image6.png
Greyscale
PNG
media_image7.png
82
100
media_image7.png
Greyscale
) taken at different distances from an object (Wang Fig. 2: Zoomed-in view is closer the object) corresponding to the 3D data (Wang Fig. 2: Dense correspondences between 2D image and 3D data), one of the images although generally zoomed-in image is expected to have more details, e.g., shape information, about the object due to higher resolution, there is no explicit teaching) as a first input image (the zoomed-in image), and selecting the other image as a second input image (the other image) (
PNG
media_image8.png
266
674
media_image8.png
Greyscale
);
estimate a first photographing position and posture corresponding to the first input image based on a degree of similarity between the virtual image information (Konishi figs. 2, 4) and the first input image, and the virtual image information (here, as explained by Wang fig. 2, 6D object pose are based on 2D-3D correspondence);
estimate a relative photographing position and posture between the first input image and the second input image (
PNG
media_image9.png
294
508
media_image9.png
Greyscale
Wang p. 16614. The relative photographing position and posture includes the disclosed translation.); and
estimate a second photographing position and posture corresponding to the second input image based on the first photographing position and posture and the relative photographing position and posture (
PNG
media_image10.png
252
504
media_image10.png
Greyscale
Wang p. 16613. The allocentric representation correspond to that of first input image; and the egocentric rotation corresponds to that of second input image, wherein the egocentric rotation is based on the allocentric rotation and relative difference between the two viewpoints.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Wang’s zooming in a viewpoint towards an object to better estimate pose of the object with Marullo in view of Konishi. One of ordinary skill in the art would be motivated to better estimate pose of an object in a photograph, when an object is sufficiently close. There will be more details/key points available about the object.
Marullo in view of Konishi and Wang does not disclose, but Hu teaches one of the images including more diverse pieces of shape information, which serves as the basis for pose estimation (
PNG
media_image11.png
378
512
media_image11.png
Greyscale
, the objectness score represents whether diverse pieces of shape/feature are strongly represented in an image.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Hu’s objectness assessment with Marullo in view of Konishi and Wang. One of ordinary skill in the art would be motivated to use a better image to assess pose information of an object. Hu states, “This simple strategy may be sufficient for object detection where one can simply choose level producing the best prediction based on the objectness scores during testing.” Hu p. 15873.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Marullo in view of Konishi as applied to Claim 1, in further view of Fletcher et al. (US 20190073792 A1) and Yin et al. (US 20200066054 A1).
Claim 8 is substantially similar to Claim 1. The rejection analyses based on Marullo in view of Konishi for Claim 1 are applied to Claim 8.
Marullo in view of Konishi does not explicitly disclose, but Fletcher teaches A photographing position and posture estimation system comprising: a photographing terminal connected to the photographing terminal so that they can communicate with each other, . . . (
“The computer module 701 is used to determine pose estimates for images capture devices such as the cameras 110 and 115. In this regard, the computer module 701 may be in communication with the cameras 110 and 115 via the network 720, as shown in FIG. 7A. The computer module 701 may be a server computer, a desktop computer or the like. Alternatively, the computer module 701 may relate to one of the cameras 110 and 115, or one of the cameras 110 and 115 may be integral to the computer module 701.” Fletcher ¶ 50.):
calculate, when the photographing terminal receives an image obtained by photographing the object as an input image, a degree of similarity . . . (Fletcher ¶ 50. The image captured is used for pose estimation based on the degree of similarity as taught by Marullo in view of Konishi.); and
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Fletcher’s with Marullo in view of Konishi. One of ordinary skill in the art would be motivated to quickly process image data by using potentially more powerful computers for at least part of the computation task.
Marullo in view of Konishi and Fletcher does not explicitly disclose, but Yin teaches return the estimated photographing position and posture to the photographing terminal (
“The camera may be controlled through the pose of the terminal device, such that the camera may capture images from a required orientation. In some embodiments, based on an acquired orientational angle of the terminal device, the terminal device may acquire an angle for which the camera needs to rotate and an orientation to which the camera needs to rotate. . . . When the pose acquired by the terminal device is the same as the predefined pose, the camera may not need to adjust the orientation, and the generated capturing control instruction may instruct the camera not to rotate for adjustment. When the pose acquired by the terminal device is different from the predefined pose, the terminal device may determine an orientation to which the camera is required to rotate and an angle for which the camera is required to rotate, based on a relative difference between the acquired pose and the predefined pose.” Yin ¶ 113.
The terminal device could also display received results to a user. “The terminal device 100 may be a head mounted display, a mobile phone, a tablet or other mobile devices. When the terminal device 100 is a head mounted display, the head mounted display may be an integrated head mounted display. The terminal device 100 may also be a head mounted display connected with an exterior electronic apparatus. The terminal device 100 may further be a smart terminal, such as a mobile phone, exteriorly connected to or plugged into a head mounted display, that is the terminal device 100 may be a processor device or a storage device of the head mounted display, and may be plugged into or connected to the head mounted display exteriorly, such that the head mounted display may display the virtual object.” Yin ¶ 25.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Yin’s camera adjustment with Marullo in view of Konishi and Fletcher. One of ordinary skill in the art would be motivated to “instruct the camera to rotate towards the required orientation and rotate the required angle, such that the orientation of the camera may correspond to the pose of the terminal device.” Yin ¶ 113. Further the terminal device could also display received results to a user. “The terminal device 100 may be a head mounted display, a mobile phone, a tablet or other mobile devices. When the terminal device 100 is a head mounted display, the head mounted display may be an integrated head mounted display. The terminal device 100 may also be a head mounted display connected with an exterior electronic apparatus. The terminal device 100 may further be a smart terminal, such as a mobile phone, exteriorly connected to or plugged into a head mounted display, that is the terminal device 100 may be a processor device or a storage device of the head mounted display, and may be plugged into or connected to the head mounted display exteriorly, such that the head mounted display may display the virtual object.” Yin ¶ 25.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Noble et al. (US 20190206084 A1): “IDENTIFYING POSE OF CAMERAS IN A SCENE,” which has the same subject matter as the claimed invention. However, Noble does not use the specific 2D-3D correspondence approach as claimed to identify the pose of the camera.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611
Prosecution Insights