DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The preliminary amendment filed on 06/24/2026 has been acknowledged.
Claims 1-10 are cancelled
Claim 11 and 16 are amended
Response to Arguments
Applicant’s arguments with respect to claims 11-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 11-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Malisiewicz et al (Malisiewicz hereinafter US 20180137642 A1.) in view of Xiang et al (Xiang hereinafter “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes”) in further view of Sequeira et al (Sequeira hereinafter US 10304237 B2)
As per claim 11
Malisiewicz teaches capturing at least one photograph, comprising depth information and image information of the object, by way of a camera of the capture device (Paragraph [0029] “cuboid detector can be used for 3D object detection as follows: fit a 3D bounding box to objects in an image (e.g., an RGB image or an RGB-Depth (RGB-D) image”), paragraph [0031] “The cuboid detector can run in real-time and perform cuboid detection with RGB images of cluttered scenes captured using a consumer-grade camera as input. A wearable display device (e.g., the wearable display system 1000 descried with reference to FIG. 10) can implement the cuboid detector and use information about the detected cuboids to generate “ and paragraph [0079] “ The image can comprise a color image “ 3D object detection and fitting a bounding box around said object is depth information in the image. Color data is image data. The capture device is the cuboid detector that uses a consumer grade camera as input); detecting the object in the image information by way of an object detection device (Paragraph [0033] “the cuboid detector can first detect the object of interest” and paragraph [0032] “The cuboid detector can learn to detect cuboids in images using a data-driven approach. The cuboid detector can assign a single label (e.g., “cuboid”) to box-like objects in a scene”) retrieving a model of the object that comprises at least one marker point, the individual situation of which in relation to the at least one planar surface is specified, based on the detection (Paragraph [0029] “perform 3D model to 2D image alignment” paragraph [0075] “The initial cuboid image location or the cuboid image location can be represented as a two-dimensional (2D)…the parameterized representation of the cuboid can include locations of a plurality of keypoints of the cuboid (e.g., a cuboid) in the image, such as six or eight vertices of the cuboid in the image” paragraph [0031] “the cuboid detector can localize the vertices of the cuboids” paragraph [0067] “The network of the cuboid detector 200 was free to use all visual evidence to localize the corners of the cuboid.” Key points and vertices of the cuboid model reasonably correspond to the claimed “Marker point” whose positions are specified relative to the cuboids planar faces. ) plausibilizing the at least one marker point on the object in the photograph (Paragraph [0033] “make coarse or initial predictions regarding the location of its vertices…performing refinement of vertices by only looking at regions with high probability of being a cuboid…cuboid detector can implement an iterative feature pooling mechanism to improve accuracy”); computing a 2-dimensional pose of the at least one planar surface based on the at least one identified marker point in the photograph (Paragraph [0029] “…3D model to 2D image alignment…”, paragraph [0076] “ locations of a plurality of keypoints of the cuboid (e.g., a cuboid) in the image, such as six or eight vertices of the cuboid in the image”, paragraph [0022] “a cuboid is represented as a tuple of eight vertices, where each vertex can be denoted by its coordinates (e.g., Cartesian x,y coordinates) in the image…a cuboid is represented by 16 parameters…Not all 16 parameters might be needed in some cases, for example… alternate cuboid representations may not include some vertices (e.g., use only six vertices) and determine the other vertices using vanishing points” This shows projective geometry in the image plane which is done by using 2D images that are projections of a 3D scene. Localizing vertices of a cuboid by its Cartesian x,y coordinates in an image and representing the cuboid using a vanishing point parameterization as well as orienting their relationships and placement on a cuboid planar surface in an image constitutes two-dimensional pose computation.
Though Malisiewicz has the functionality for computing 6D positioning (Paragraph [0090] “ The cuboid detector can detect cuboid vertices in different views and determine a final cuboid configuration based on a score from the HOG classifier, 2D vertex displacement “ , Paragraph [0029] “An embodiment of a cuboid detector can be used for 3D object detection as follows: fit a 3D bounding box to objects in an image (e.g., an RGB image or an RGB-Depth (RGB-D) image)”Paragraph [0059] “additional or different parameters can be used. [0060] (X, Y, Z)—Coordinates of the center of the cuboid in 3D [0061] (L, W, H)—Dimensions of the cuboid [0062] (θ, ψ, φ)—3 angles of rotation of the cuboid “ ), Malisiewicz will not be relied upon for the preamble, selecting a set of points, which is assigned to the at least one planar surface of the object, from the depth information of the photograph and computing the six-dimensional situation of the object by aligning the set of points with the computed 2-dimensional pose, the situation describing a position and an orientation of the object relative to the capture device.
Xiang teaches A method for determining a six-dimensional situation of an object having at least one planar surface relative to a capture device (Section 3 PoseCNN “ Given an input image, the task of 6D object pose estimation is to estimate the rigid transformation from the object coordinate system O to the camera coordinate system C… rigid transformation here consists of an SE(3) transform containing a 3D rotation R and a 3D translation T, where R specifies the rotation angles around the X-axis, Y-axis and Z axis of the object coordinate system O, and T is the coordinate of the origin of O in the camera coordinate system C.” Figure 5 shows various objects under the method of detection with at least one planar surface. ) selecting a set of points, which is assigned to the at least one planar surface of the object, from the depth information of the photograph (Abstract: “When using depth data to further refine the poses, our approach achieves state-of-the-art results”, Introduction: “we use depth images in the Iterative Closest Point (ICP) algorithm for pose refinement”, Experiment section D Baselines : “Semantic labels from the network are used to crop the observed points from the depth image.”); and computing the six-dimensional situation of the object by aligning the set of points with the computed 2D pose, the situation describing a position and an orientation of the object relative to the capture device. (Experiment section D Baselines: “. “The 6D pose estimated from our network can be refined when depth is available. We use the Iterative Closest Point (ICP) algorithm to refine the 6D pose. Specifically, we employ ICP with projective data association and a point-plane residual term.” , Section 3 PoseCNN: “Given an input image, the task of 6D object pose estimation is to estimate the rigid transformation from the object coordinate system O to the camera coordinate system C.”
Malisiewicz nor Xiang teach wherein the set of points is selected on a random basis.
Sequeira teaches wherein the set of points is selected on a random basis. (Paragraph (179) “Starting from this guess, each iteration of the ICP algorithm creates n random sets of points, S, where each set s.sup.(j)∈S contains k randomly selected points from each bin (typically k=1 For each one of these points, one computes the associated position in world coordinates, p.sub.i.sup.W, using {circumflex over (Γ)}.sub.t and its corresponding nearest plane in the map…once all correspondences in each set are solved, the rigid transformation T.sup.(j)=[R.sup.(j)t.sup.(j)] “ is computed for each set. Sequeira therefore expressly teaches forming set of points for ICP alignment in which the points in each set are randomly selected.)
In a combined teaching Malisiewicz teaches detecting the object and identifying its geometric features and image-based pose information. Xiang teaches selecting observed points associated with the detected object from depth information and using those points in an ICP alignment to determine and refine the 6D pose of the object. Sequeira adds the concept that when performing ICP alignment using observed points, the points used for the alignment may be selected on a random basis by creating random sets containing randomly selected points, with the resulting correspondences used to compute a rigid transformation including rotation and translation.
Accordingly at the time this invention was effectively filed, a person of ordinary skill in the art would have found it obvious to modify the depth based ICP pose refinement of Xiang as it is applied to the object detection and pose determination method of Malisiewicz, to select the points used for ICP alignment on a random basis as taught by Sequeira. Xiang already allows observed depth points and point plane ICP to refine the objects 6D pose, while Sequeira teaches a known implementation of ICP in which random sets containing randomly selected observed points are used to establish point plane correspondences and determine the rigid transformation. Malisiewicz already allows the detected object and its localized geometric features (which includes cuboid vertices and image based pose information to which Xiang’s depth based 6D pose refinement is applied. Applying Sequeira’s random point selection technique to Xiang’s ICP refinement within the Malisiewicz/Xiang portion of the system would result in the detected object and its geometric features identified by Malisiewicz being associated with the points in Xiang. Those depth points being randomly selected according to Sequeira for ICP alignment and determination of the object’s 6D position and orientation relative to the camera. A person of ordinary skill would have had reason to apply Sequeira’s random point selection in Xiang’s ICP process because Sequeira shows using random sets in correspondence with identifying and removing outliers. This provides an efficient determination of the transformation from the observed points.
As per claim 12
Malisiewicz , Xiang and Sequeira cover the claim limitations previously rejected in claim 11’s 103 rejection. See claim 11’s 103 rejection.
Malisiewicz teaches performing a plausibility check on at least one identified marker point before computing of the orientation and/or the situation (paragraph [0033] “the cuboid detector can first detect the object of interest and then make coarse or initial predictions regarding the location of its vertices. The cuboid can utilize the coarse or initial predictions as an attention mechanism, performing refinement of vertices by only looking at regions with high probability of being a cuboid. In some embodiments, the cuboid detector can implement an iterative feature pooling mechanism to improve accuracy. The cuboid detector can combine cuboid-related losses and or implement alternate parametrizations to improve accuracy.”
In regards to “before computing of the orientation and/or the situation.” Malisiewicz “first detect the object of interest and then make coarse or initial predictions regarding the location of its vertices” then “performs refinement of vertices” after. Only after that in the Malisiewicz/Xiang/Sequeira modified methodology, will Xiang’s “task of 6D object pose estimation ... to estimate the rigid transformation…of an SE(3) transform containing a 3D rotation R and a 3D translation T, where R specifies the rotation angles around the X-axis, Y-axis and Z axis of the object coordinate system O, and T is the coordinate of the origin of O in the camera coordinate system C” take place. A person of ordinary skill in the art knows would know to orient the steps this way in the modified system to ensure the plausibility check is performed on marker points before the marker points are relied upon for pose orientation and 6D computation.
Therefore, it would have been obvious to a person of ordinary skill in the art at the time this invention was effectively filed to perform Malisiewicz’s plausibility check on the identified marker points in the combined methodology before computing Xiang’s orientation and or 6D situation because the later pose computation depends on the reliability of the identified marker points as a sort of anchor. Malisiewicz teaches refining the predicted vertex locations by looking only at regions with high probability of being a cuboid and using feature pooling to improve accuracy. A person of ordinary skill in the art would know that doing this refinement before later orientation and 6D pose computation would reduce the surfacing of false key point locations when the later pose is estimated. This improves stability and accuracy of the final position relative to the camera. Unreliable marker point predictions are filtered or refined before they are actually used to create the final pose. This improves the later accuracy of the 2D orientation estimate and the final 6D estimate.
As per claim 13
Malisiewicz Xiang and Sequeira cover the claim limitations previously rejected in claim 12’s 103 rejection. See claim 12’s 103 rejection.
Malisiewicz teaches the plausibility check results in a geometry, a confidence, and/or a consistency of the at least one identified marker point being determined. (Paragraph [0033] “ the cuboid detector can first detect the object of interest and then make coarse or initial predictions regarding the location of its vertices. The cuboid can utilize the coarse or initial predictions as an attention mechanism, performing refinement of vertices by only looking at regions with high probability of being a cuboid. In some embodiments, the cuboid detector can implement an iterative feature pooling mechanism to improve accuracy. The cuboid detector can combine cuboid-related losses and or implement alternate parametrizations to improve accuracy.” “Confidence” is shown when Malisiewicz uses regions with “high probability of being a cuboid” because the probability-based refinement process determines whether the predicted marker point (vertices) is reliable for downstream use. “Geometry” is taught because the plausibility check is performed on predicted vertex locations of a cuboid and those same locations are refined based on if they are actually reliable. The refinement determines whether the identified marker points truly conform to the targeted or expected cuboid shape. “Consistency” is showcased by the fact that the vertex predictions are checked against a sort of cuboid hypothesis which is created by the detector only looking at regions with a “high probability of being a cuboid”. This means the detector determines whether the identified markers are consistent with the object that has been detected.)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to use Malisiewicz concept of refinement/plausibility checking within the Malisiewicz/Xiang/Sequeira methodology so that the check results in determining a geometry, a confidence and a consistency of the identified marker points before those marker points are used for the downstream 6D orientation and 6D situation computation. One of ordinary skill in the art would have been motivated to do this implementation because Malisiewicz expressly teaches using probability based refinement and feature pooling to improve the reliability and accuracy of the vertex locations. A skilled practitioner would have understood that the later pose estimation is dependent on the reliability and fidelity as well as the geometric fidelity of the vertices (markers). By determining whether the marker points are probable, geometrically conforming and consistent with the high probability cuboid before using in later computations, the modified methodology can reduce pushing incorrect or erroneous locations down the pipeline and into the final orientation of 6D pose. A person of ordinary skill in the art gains the advantage of a method that filters and refines unreliable marker point predictions before they are used for final pose estimations. This improves efficiency and accuracy of the final position and orientation determination.
As per claim 14
Malisiewicz Xiang and Sequeira cover the claim limitations previously rejected in claim 11’s 103 rejection. See claim 11’s 103 rejection.
Malisiewicz teaches machine learning methods are used for detecting the object and/or for identifying the at least one marker point. (Paragraph [0034] “The cuboid detector 200 can include one or more of the following components: a convolutional layer 204 (also referred to herein as a CNN tower), a Region Proposal Network (RPN) 208, at least one pooling layer 212, or one or more fully connected layers 216 (e.g., a regional CNN (R-CNN) regressor (or classifier)). The pooling layer 212 and the fully connected layers 216 can implement iterative feature pooling, which refines cuboid keypoint locations. The R-CNN can be a Faster R-CNN. The cuboid detector 200 can implement a deep cuboid detection pipeline” )
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to use a machine learning method for detecting the object and for identifying the marker points in the combined Malisiewicz/Xing/Sequeira methodology. Malisiewicz expressly shows deep learning based cuboid detector that performs both cuboid detection and keypoint localization while Xing uses a learned pose estimation pipeline whose output is further refined using depth information. A person of ordinary skill in the art would have understood that using machine learning for the detection and marker point identification stages would provide reliable object recognition and keypoint extraction even in cluttered scenes. This supplies high end inputs for the downstream 6D pose computation and alignment stages. The modified methodology gives the advantage of improved and more efficient visual recognition stages which then improves the accuracy of the pose and 6D situation computation stages.
As per claim 15
Malisiewicz, Xiang, and Sequeira cover the claim limitations previously rejected in claim 14’s 103 rejection. See claim 14’s 103 rejection.
Malisiewicz teaches machine learning methods comprise deep learning and/or at least one convolutional neural network. (Paragraph [0107] “In a 1st aspect, a system for cuboid detection and keypoint localization is disclosed. The system comprises: non-transitory memory configured to store: executable instructions, an image for cuboid detection, and a cuboid detector comprising: a plurality of convolutional layers and non-convolutional layers of a first convolutional neural network (CNN) for generating a convolutional feature map from the image, a region proposal network (RPN) comprising a second CNN for determining, using the convolutional feature map, at least one region of interest (RoI) comprising a cuboid at a cuboid image location of the image, and a pooling layer and at least one regressor layer for determining, using the convolutional feature map”
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to implement a machine learning method that uses at least one convolutional neural network as taught by the Malisiewicz /Xiang/Sequeira methodology. A person of ordinary skill in the art would have understood that using deep learning and convolutional neural network processing in the beginning would provide efficient extraction of the object and the marker points from the image data. This in turn supplies reliable inputs for the downstream 6D computations. The use of a CNN reduces data complexity via pooling layers. The overall object detection is improved even more so If the image is noisy for example.
As per claim 16
Malisiewicz Xiang and Sequeira cover the claim limitations previously rejected in claim 11’s 103 rejection. See claim 11’s 103 rejection.
Xiang teaches the set of points is selected on a plane equation (Experiment section, D. Baselines “We use the Iterative Closest Point (ICP) algorithm to refine the 6D pose. Specifically, we employ ICP with projective data association and a point-plane residual term… The residual for each pixel is then the smallest distance from the observed point in 3D to the plane defined by the rendered point in 3D and its normal. Points with residuals above a specified threshold are rejected and the remaining residuals are minimized using gradient descent. Semantic labels from the network are used to crop the observed points from the depth image.” This shows selection of points based on a plane equation. This is because the observed dept points are evaluated relative to a plane defined by the rendered point and the normal point. Then points are retained or dismissed based on the residual that corresponds to that plane. The depth point is selected according to a point to plane relationship defined by the plane. )
Accordingly, it would have been obvious to a person of ordinary skill in the art at the time the invention was effectively filed would have found it obvious to select points based on a plane equation in the Malisiewicz /Xing/Sequeira methodology. This is because in the Malisiewicz /Xing/Sequeira methodology, once the object and its pose features are identified, the set of points (which were selected at random through Sequeira’s methodology) taken from the depth satisfy the point to plane criterion used in the pose refinement stage. A person of ordinary skill in the art would have understood that in a methodology like the Malisiewicz /Xing/Sequeira modified methodology that identifies a planar faced object and then proceeds to determine the object’s position and orientation, randomly selecting depth points as long as the random set is conforms with the modeled plane of the object surface would improve the overall quality of the point set used for alignment simply by excluding inconsistent points yet preserving points that correspond to the targeted planar surface. The advantage gained is that the depth point set used for final pose alignment is filtered according to the plane correlated with the object’s surface. This improves the alignment accuracy by reducing the influence of non-surface or inconsistent depth points.
As per claim 17
Malisiewicz Xiang and Sequeira cover the claim limitations previously rejected in claim 11’s 103 rejection. See claim 11’s 103 rejection.
Malisiewicz teaches at least one of the steps is performed by way detecting the object and/or identifying and/or computing the orientation is carried out based on pattern comparison. (Paragraph [0005] “a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in images”, paragraph [0033] “the cuboid detector can first detect the object of interest and then make coarse or initial predictions regarding the location of its vertices.”, paragraph [0029] “ perform 3D model to 2D image alignment.” Paragraph [0076] “the parameterized representation of the cuboid can include locations of a plurality of keypoints of the cuboid (e.g., a cuboid) in the image, such as six or eight vertices of the cuboid in the image.” This shows that detecting the object is carried out by comparing image content against cuboid patterns and features in the deep cuboid detector. Identifying the marker points is carried out by localizing cuboid vertices in the image. The computation of orientation is done by using the detected key point pattern together with 3D model to 2D alignment. Essentially the observed image pattern is compared against the learned pattern of the cuboid in order to detect the cuboid and localize the vertices.
It would have been obvious to a person skilled in the art at the time this invention was effectively filed to implement the detection, marker point identification and orientation processing of the Malisiewicz/Xing/Sequeira methodology based on pattern comparison because Malisiewicz expressly teaches a deep cuboid detector for cuboid detection and key point localization in images and further teaches a 3D model to 2D image alignment using the detected key point pattern. A person of ordinary skill in the art would have understood that detecting the object and identifying its marker points by comparing observed image features against learned model cuboid patterns provides heightened fidelity in regards to geometric anchoring for later pose estimation. Supplying those patterns derived positions to the 6D pose estimation pipeline of Xing would improve the ability of the overall system to determine objects position and orientation. The modified methodology enables pattern based detection and key point identification that provide a structured and reliable front end for alter downstream pose estimation improving efficiency of the downstream 6D situation computation.
As per claim 18
Malisiewicz ,Xiang and Sequeira cover the claim limitations previously rejected in claim 11’s 103 rejection. See claim 11’s 103 rejection.
Malisiewicz teaches the 2-dimensional pose is computed by way of projective geometry, (Figure 6, Paragraph [0076] “ the parameterized representation of the cuboid comprises a vanishing point parameterization.” Then explains in paragraph [0069] that “Another re-parameterization uses locations of the two vanishing points and the slopes of six lines which will form the edges of the cuboid…The intersection points of these six lines would give the vertices of the cuboid in this example.” Furthermore, a 2D pose is localized keypoints of a three-dimensional object within a single photo or video frame. Malisiewicz cuboid detector performs “3D model to 2D image alignment” and determines cuboid parameters from image data. In paragraph [0022] Malisiewicz states that “ a cuboid is represented as a tuple of eight vertices, where each vertex can be denoted by its coordinates (e.g., Cartesian x,y coordinates) in the image…a cuboid is represented by 16 parameters…Not all 16 parameters might be needed in some cases, for example… alternate cuboid representations may not include some vertices (e.g., use only six vertices) and determine the other vertices using vanishing points.” These passages support using vanishing point parameterization, a fundamental concept of projective geometry, to compute claimed 2D pose from the image.)
Malisiewicz teaches the front-end portion of 2D pose in the Malisiewicz/Xiang/Sequeira. It detects the cuboid object, localizes its key points/vertices and uses a vanishing point parameterization with line slopes and line intersections to represent the cuboid in the image plane. Those teachings supply the geometric basis for computing the 2D pose of the planar surface in the image domain. Xing then enables the downstream 6D pose side of the method. Xiang defines 6D pose estimation as estimating the “rigid transformation from the object coordinate system O to the camera coordinate system C” where the transform contains “a 3D rotation R and a 3D translation T” and it refines that pose using depth including that the “Semantic labels from the network are used to crop the observed points from the depth image” Sequeira randomizes the points.
Accordingly in the combined methodology, the 2D pose is first supplied by Malisiewicz using projective geometric image relationships for the cuboid. That pose then feeds the later depth based six-dimensional situation computation of Xiang. Therefore, it would have been obvious to one of ordinary skill in the art at the time this invention was effectively filed to compute the 2D pose in the combined Malisiewicz/Xiang/Sequeira methodology by way of projective geometry. A person of ordinary skill in the art would have understood that these projective geometric relationships enable a direct and reliable path to derive the image plane pose/orientation of an object with planes before performing the 6D pose refinement of Xing. Using projective geometry in the upstream processes would predictably allow for a consistent 2D pose estimation that could ten be used as an input for the 3D pose estimation improving the fidelity of the 6D output.
As per claim 20
Malisiewicz , Xiang, and Sequeira cover the claim limitations previously rejected in claim 11’s 103 rejection. See claim 11’s 103 rejection.
The Malisiewicz teaches in paragraph [0133] “hardware processor in communication with the non-transitory memory, the hardware processor programmed by the executable instructions to:” and in paragraph [0204] states “it should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.” By extension through modification the Malisiewicz /Xiang/ Sequeira methodology teaches the claim limitations of claim 11.
Accordingly, it would have been obvious to one of ordinary skill in the art at the time the invention was effectively filed to provide the Malisiewicz /Xiang/Sequeira method in the form of a computer product comprising a non-transitory computer readable medium storing program code executable by a electronic computing device. This is because both references already disclose computer executed image processing and pose estimation pipelines that are done through software that drives neural network inferences and pose refinement. A person of ordinary skill in the art would have understood that storing those instructions on a non-transitory computer readable medium for execution by a device is the conventional, traditional and expected avenue for deploying computer vision and would need to do so for repeatability of method. The full detection and pose pipeline can be packaged and executed as a deployable software product on a computing device allowing for automated and repeatable execution of methods.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Malisiewicz et al (Malisiewicz hereinafter US 20180137642 A1.) in view of Xiang et al (Xiang hereinafter “PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes”) in further view of in further view of Sequeira et al (Sequeira hereinafter US 10304237 B2) in further view of Höfer, et al (Höfer hereinafter “Object Detection and Autoencoder-Based 6D Pose Estimation for Highly Cluttered Bin Picking”)
Malisiewicz, Xiang and Sequeira cover the claim limitations previously rejected in claim 11’s 103 rejection. See claim 11’s 103 rejection.
The Malisiewicz/Xiang/Sequeira method does not cover when capturing the photograph, the orientation of the camera in relation to a logistics facility is specified, and/or the method is used in vehicle production.
Höfer teaches when capturing the photograph, the orientation of the camera in relation to a logistics facility is specified (Abstract: “Bin picking is a core problem in industrial environments and robotics, with its main module as 6D pose estimation.” Höfer states in the introduction that “Bin picking is a major automation task with various applications in industrial sectors. The core starting problem of this work is the 6D pose estimation of instances. To tackle this problem, an RGB-D or depth camera is usually installed on top of the bin.” In Höfer’s figure 1 we can see that the camera is oriented above a cluttered bin and in section 3.1.3 Test dataset for object detection “To evaluate the object detector, we captured 50 real images per object model with more than 100 instances in the bin. The images were taken using a Microsoft Azure Kinect camera mounted at a height of 30cm over the bin (see Fig. 4” showing a real industrial scenario.)
Accordingly, it would have been obvious to one of ordinary skill in the art at the time this invention was effectively filed to further modify the Malisiewicz/Xiang/Sequeira methodology of object detection and 6D pose estimation in the industrial bin picking environment shown by Höfer, such that when capturing the photograph, the orientation of the camera in relation to the logistics facility is specified. One of ordinary skill in the art would have been motivated to do so because Höfer teaches that bin pocking in industrial environments and robotics has 6D estimation as its main module while Xing teaches that 6D pose is determined as a rigid transformation relative to the camera coordinate system. A person of ordinary skill in the art would understand that, to deploy the pose estimation pipeline of the Malisiewicz/Xiang/Sequeira in the industrial logistic environment of Höfer for robotic picking, the camera must have a known orientation relative to the facility so that the camera relative pose estimate can be used in the industrial environment. The advantage is that the known camera orientation relative to the logistics facility allows the detected object pose to be used more reliably in handling. The camera relative 6D estimate can be consistently related its surroundings for industrial operations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights