Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The United States Patent & Trademark Office appreciates the application that is submitted by the inventor/assignee. The United States Patent & Trademark Office reviewed the following application and has made the following comments below.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on XXX is considered and attached.
Priority
This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of:
EP21208064.2, filed in United Kingdom on 11/12/2021.
PCT/GB2022/052832, filed in United Kingdom on 11/09/2022.
Claim Status
Claims 1-25 are rejected under 35 U.S.C. 112(b).
Claims 1-3, 22 and 26 are rejected under 35 U.S.C. 102 in view Birchfield.
Claims 4-5, 8-11, 15-17 and 23-25 are rejected under 35 USC § 103:
Claims 4-5 are rejected over Birchfield in view of Zhou.
Claims 8-9 are rejected over Birchfield in view of Kim.
Claims 10-11 are rejected over Birchfield in view of Kim, and further in view of Gao.
Claims 15-17 are rejected over Birchfield in view of Han.
Claim 23 is rejected over Birchfield in view of Srinivasan, and further in view of Patel.
Claim 24 is rejected over Birchfield in view of Patel.
Claim 25 is rejected over Birchfield in view of Sheorey.
Claims 12-14 are objected.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-25 (include dependent claims) are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The examiner strongly suggested that appropriate corrections be made to clarify the claim scope.
With respect to Claim 1, the claim recites the following, each of which renders the claim indefinite:
“the dimensions of an article” on line 1 (unclear antecedent basis).
With respect to Claim 4, the claim recites the following, each of which renders the claim indefinite:
“the camera” on line 2 (unclear antecedent basis).
“the floor” on line 2 (unclear antecedent basis).
With respect to Claim 10, the claim recites the following, each of which renders the claim indefinite:
“the width” on line 3 (unclear antecedent basis).
With respect to Claim 23, the claim recites the following, each of which renders the claim indefinite:
“the total cabin storage capacity” on line 4 (unclear antecedent basis).
With respect to Claim 24, the claim recites the following, each of which renders the claim indefinite:
“the remaining cabin storage” on line 1 (unclear antecedent basis).
Claims 2-3, 8-9, 11-17, 22 and 24-25 are also rejected due to their dependence on rejected independent claim 1.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 22 and 26 are rejected under 35 U.S.C. 102 (a)(2) as being anticipated by Birchfield et al. (US-20220277472-A1, filed 09/09/2021, hereinafter Birchfield)
CLAIM 1
Regarding Claim 1, Birchfield teaches a method for measuring the dimensions of an article (Birchfield, abstract: “Apparatuses, systems, and techniques to determine a pose and relative dimensions of an object from an image.”), the method comprising: obtaining image data associated with an article (Birchfield, ¶ [0061]: “obtains an RGB (red-green-blue) image depicting an object”);
identifying, based on the image data, a plurality of 2D pixel locations associated with the article (Birchfield, ¶ [0082 and 0085]: “determines pixel coordinates of an input image 104 that correspond to a center of an object, and utilizes width and height values from a 2D bounding box size 114 corresponding to the pixel coordinates to determine a size of a bounding box of the object … may utilize a keypoint heatmaps 118 and a keypoint offsets 120 to determine pixel coordinates of each keypoint of a 3D bounding box of an object depicted in an input image 104. ”);
calculating corresponding 3D coordinates for each of the 2D pixel locations (Birchfield, ¶ [0060]: “A 6-DoF pose may refer to a three-dimensional (3D) position and orientation of an object…”; ¶ [0093-0097]: “A PnP algorithm may process keypoint locations calculated by a 2D keypoint output decoding, a relative dimensions, and a camera intrinsics to output a 6-DoF pose and dimensions. A PnP algorithm may output a set of values indicating a 6-DoF pose and dimensions” Birchfield teaches using PnP algorithm to calculate 3D coordinate of an object based on its 2D pixel coordinates); and
determining the dimensions of the article based on the 3D coordinates (Birchfield, ¶ [0093-0097]: “A PnP algorithm may process keypoint locations calculated by a 2D keypoint output decoding, a relative dimensions, and a camera intrinsics to output a 6-DoF pose and dimensions…A 6-DoF pose may be indicated by a bounding cuboid (e.g., coordinates of vertices of the bounding cuboid). A 6-DoF pose may be relative to a position and/or orientation of a camera, or any suitable reference point or plane. Relative dimensions may be indicated by one or more values that correspond to a ratio of width to height to length of a bounding cuboid of an object”; ¶ [0134-0136]: “calculates an absolute scale based at least in part on the one or more relative dimension values. Absolute scale of an object, also referred to as absolute dimensions, may refer to dimensions of the object in the real-world. In some examples, an absolute scale of an object corresponds to dimensions of a bounding cuboid of the object in the real world.” Birchfield teaches calculate dimensions of an object in real-world system, based on 3D bounding box of the object).
CLAIM 2
Regarding Claim 2, Birchfield teaches the method of Claim 1. In addition, Birchfield teaches the 3D coordinates are calculated using a Perspective-n-Point (PnP) algorithm (222,530) based on camera calibration data. (Birchfield, ¶ [0060]: “A 6-DoF pose may refer to a three-dimensional (3D) position and orientation of an object…”; ¶ [0093-0097]: “A PnP algorithm may process keypoint locations calculated by a 2D keypoint output decoding, a relative dimensions, and a camera intrinsics to output a 6-DoF pose and dimensions. A PnP algorithm may output a set of values indicating a 6-DoF pose and dimensions. A 6-DoF pose and dimensions may be a set of data indicating a 6-DoF pose and relative dimensions of a bounding cuboid of an object depicted in an input image”
CLAIM 3
Regarding Claim 3, Birchfield teaches the method of Claim 2. In addition, Birchfield teaches the dimensions of the article are determined based on a calculated scaling factor (Birchfield, ¶ [0111]: “Additional metadata may include camera poses, sparse point clouds, and surface planes, with the latter assuming that the object rests on the ground plane, which may yield an absolute scale factor”) and the 3D coordinates. (Birchfield, ¶ [0134-0136]: “calculates an absolute scale based at least in part on the one or more relative dimension values. Absolute scale of an object, also referred to as absolute dimensions, may refer to dimensions of the object in the real-world. In some examples, an absolute scale of an object corresponds to dimensions of a bounding cuboid of the object in the real world.”) (The Examiner notes the absolute scale (absolute dimensions) of the object is determined based on an absolute scale factor and the relative dimensions, which obtained from the 3D bounding box)
CLAIM 22
Regarding Claim 22, Birchfield teaches the method of Claim 1. In addition, Birchfield teaches each of the plurality of 2D pixel locations is identified using a neural network. (Birchfield, ¶ [0081-0082]: “ a system for object for pose estimation 102 (e.g., via a 2D keypoint output decoding 128) determines pixel coordinates of an input image 104 that correspond to a center of an object, and utilizes width and height values from a 2D bounding box size 114 corresponding to the pixel coordinates to determine a size of a bounding box of the object”; ¶ [0078]: “A group 108 may implement one or more neural network layers and/or processes that process a feature map to output an object center heatmap 110, an object center offset 112, and a 2D bounding box size 114”; see system 102 and group 108 in FIG. 1)
CLAIM 26
Regarding claim 26, Birchfield teaches a system (100) for measuring the dimensions of an article (Birchfield, abstract: “Apparatuses, systems, and techniques to determine a pose and relative dimensions of an object from an image.”), the system comprising:
a camera (110) (Birchfield, ¶ []: “A camera intrinsics 132 may be a set of data indicating intrinsics of an image and/or video capturing device that was utilized to capture an input image 104”) configured to obtain image data associated with an article (Birchfield, ¶ [0061]: “obtains an RGB (red-green-blue) image depicting an object”);
a neural network (112) (Birchfield, ¶ [0081-0082]: “ a system for object for pose estimation 102 (e.g., via a 2D keypoint output decoding 128) determines pixel coordinates of an input image 104 that correspond to a center of an object, and utilizes width and height values from a 2D bounding box size 114 corresponding to the pixel coordinates to determine a size of a bounding box of the object”; ¶ [0078]: “A group 108 may implement one or more neural network layers and/or processes that process a feature map to output an object center heatmap 110, an object center offset 112, and a 2D bounding box size 114”; see system 102 and group 108 in FIG. 1) configured to identify, based on the image data, a plurality of 2D pixel locations associated with the article (Birchfield, ¶ [0082 and 0085]: “determines pixel coordinates of an input image 104 that correspond to a center of an object, and utilizes width and height values from a 2D bounding box size 114 corresponding to the pixel coordinates to determine a size of a bounding box of the object … may utilize a keypoint heatmaps 118 and a keypoint offsets 120 to determine pixel coordinates of each keypoint of a 3D bounding box of an object depicted in an input image 104. ”); and
a processor (Birchfield, ¶ [0124]: “ In at least one embodiment, code is stored on a computer-readable storage medium in form of a computer program comprising a plurality of computer-readable instructions executable by one or more processors”) configured to calculate corresponding 3D coordinates for each of the 2D pixel locations (Birchfield, ¶ [0060]: “A 6-DoF pose may refer to a three-dimensional (3D) position and orientation of an object…”; ¶ [0093-0097]: “A PnP algorithm may process keypoint locations calculated by a 2D keypoint output decoding, a relative dimensions, and a camera intrinsics to output a 6-DoF pose and dimensions. A PnP algorithm may output a set of values indicating a 6-DoF pose and dimensions” Birchfield teaches using PnP algorithm to calculate 3D coordinate of an object based on its 2D pixel coordinates), and further configured to determine the dimensions of the article based on the 3D coordinates (Birchfield, ¶ [0093-0097]: “A PnP algorithm may process keypoint locations calculated by a 2D keypoint output decoding, a relative dimensions, and a camera intrinsics to output a 6-DoF pose and dimensions…A 6-DoF pose may be indicated by a bounding cuboid (e.g., coordinates of vertices of the bounding cuboid). A 6-DoF pose may be relative to a position and/or orientation of a camera, or any suitable reference point or plane. Relative dimensions may be indicated by one or more values that correspond to a ratio of width to height to length of a bounding cuboid of an object”; ¶ [0134-0136]: “calculates an absolute scale based at least in part on the one or more relative dimension values. Absolute scale of an object, also referred to as absolute dimensions, may refer to dimensions of the object in the real-world. In some examples, an absolute scale of an object corresponds to dimensions of a bounding cuboid of the object in the real world.” Birchfield teaches calculate dimensions of an object in real-world system, based on 3D bounding box of the object).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Birchfield in view of Zhou et al. (Zhou, Dingfu, Yuchao Dai, and Hongdong Li. "Ground-plane-based absolute scale estimation for monocular visual odometry." IEEE, published 2019, hereinafter Zhou).
CLAIM 4
Regarding Claim 4, Birchfield teaches the method of Claim 3. In addition, Birchfield teaches the scaling factor is calculated based on camera position to a ground plane (Birchfield, ¶ [0111]: “Additional metadata may include camera poses, sparse point clouds, and surface planes, with the latter assuming that the object rests on the ground plane, which may yield an absolute scale factor”); a relative height of the article and a relative depth of the article. . (Birchfield, ¶ [0134-0136]: “calculates an absolute scale based at least in part on the one or more relative dimension values. Absolute scale of an object, also referred to as absolute dimensions, may refer to dimensions of the object in the real-world. In some examples, an absolute scale of an object corresponds to dimensions of a bounding cuboid of the object in the real world.”, ¶ [0103]: “… In some examples, a width value, a height value, and a length value of relative dimensions represent values of a width, a height, and a length, respectively, of a 3D bounding box relative to each other”)
Birchfield does not explicitly disclose the scaling factor is calculated based on a predetermined height between the camera and the floor.
Zhou is in the same field of art of absolute scale estimation for monocular visual odometry. Further, Zhou teaches the scaling factor is calculated based on a predetermined height between the camera and the floor (Zhou, page 3-4, section III. CAMERA HEIGHT BASED SCALE ESTIMATION: “Assuming that an object’s length is b based on the 3D reconstruction, the scale factor is defined as s = b/b, where b is the ground true length of this object. Absolute scale estimation aims at recovering this coefficient s. Camera height is commonly used for scale estimation. Usually, the camera is fixed on a platform and its height (the distance from camera principle center to the ground plane) is unchanged during a certain amount of time. Assuming the ground surface right in front of the camera is flat, the scale can be recovered according to this height information” Zhou teaches obtaining scale factor using height of a fixed camera to the ground surface).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Birchfield by incorporating camera to ground plane model that is taught by Zhou, to make a monocular system that calculate scale from camera height; thus, one of ordinary skilled in the art would be motivated to combine the references since it’s a simple substitution, Birchfield teaches using camera position and ground plane information to obtain scale factor, and Zhou teaches detailed approaches for such method (Zhou, page 3-4, section III. CAMERA HEIGHT BASED SCALE ESTIMATION).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 5
Regarding Claim 5, the combination of Birchfield and Zhou teaches the method of Claim 4. In addition, the combination of Birchfield and Zhou teaches wherein the relative height of the article and the relative depth of the article are determined based on the calculated 3D coordinates (Birchfield, ¶ [0102-0103]: “A 3D bounding box may be indicated by coordinates of vertices of the 3D bounding box…Relative dimensions of a 3D bounding box may refer to a set of values that indicate sizes of dimensions of the 3D bounding box (e.g., width, length, and/or height of the 3D bounding box) relative to each other. In an embodiment, relative dimensions of a 3D bounding box comprise a width value, a height value, and a length value, which form a ratio of width to height to length (e.g., the width value to the height value to the length value) of the 3D bounding box”.), wherein the height and depth of the article are relative to the width of the article. (Birchfield, ¶ [0103]: “… In some examples, a width value, a height value, and a length value of relative dimensions represent values of a width, a height, and a length, respectively, of a 3D bounding box relative to each other”)
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Birchfield in view of Kim et al. (US-20070253618-A1, hereinafter Kim).
CLAIM 8
In regards to Claim 8, Birchfield teaches the method of Claim 1.
Birchfield does not explicitly disclose determining (223,540) a plurality of corresponding 2D coordinates based on the calculated 3D coordinates.
Kim is in the same field of art of 3D object reconstruction from 2D image. Further, Kim teaches determining (223,540) a plurality of corresponding 2D coordinates based on the calculated 3D coordinates. (Kim, ¶ [0092-0096]: “the 3D points obtained from the SAM initialization are re-projected on a 2D plane using a camera projection matrix”, see reconstructed text below)
PNG
media_image1.png
356
901
media_image1.png
Greyscale
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Birchfield by incorporating method to determine reprojection error that is taught by Kim, to make a 3D reconstruction system that can perform optimization; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to increase the accuracy of such system (Kim, ¶ []: “An object extraction may involve finding a solution minimizing an error measure, and after an initial solution is obtained, the initial solution may be optimized in order to find an accurate solution”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 9
Regarding Claim 9, the combination of Birchfield and Kim teaches the method of Claim 4. In addition, the combination of Birchfield and Kim teaches comparing the plurality of 2D coordinates with the plurality of 2D pixel locations associated with the article to determine an error (224). (Kim, ¶ [0092-0096]: “From the distances between feature points obtained from an image and re-projected points when the 3D points obtained from the SAM initialization are re-projected on a 2D plane using a camera projection matrix, the quality of the pose may be evaluated through a distance function f(P) …”, see reconstructed text below)
PNG
media_image2.png
356
901
media_image2.png
Greyscale
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Birchfield in view of Kim, and further in view of Gao et al. (US-20200065988-A1, hereinafter Gao).
CLAIM 10
In regards to Claim 10, the combination of Birchfield and Kim teaches the method claim 9.
The combination of Birchfield and Kim does not explicitly disclose identifying (225) a discrete range of acceptable values.
Gao is in the same field of art of measuring object’s dimension. Further, Gao teaches identifying (225) a discrete range of acceptable values. (Gao, ¶ [0044]: “… FIG. 3L, provides an example of airline baggage size requirements and a visual indicator if the measured bag is within the requirements for a given airline. The airline baggage size requirements may be modified to compensate for inaccuracies in the measured object size. This list of airlines may include a combination of a static list of data and/or a dynamic list of airline data retrieved from a server, allowing for an automatic adjustment of the airline list to better identify airlines in which the object will fit”, see modified FIG. 3L below. Gao teaches a list of
PNG
media_image3.png
577
694
media_image3.png
Greyscale
acceptable dimension values)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Birchfield and Kim by incorporating the mobile device and program and that is taught by Kim, to make a mobile device with highly intuitive user interface for measuring object’s dimension; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to improve user’s experience (Kim, ¶ [0044]: “A number of user interfaces can be provided to facilitate a user's ability to capture image data and determine the size of objects. FIGS. 3A-Q illustrate a variety of user interface screenshots for measuring objects, displaying tutorials and other notifications, and a variety of input elements for obtaining input data during the measurement processes … FIG. 3L, provides an example of airline baggage size requirements and a visual indicator if the measured bag is within the requirements for a given airline … overlays of an acceptable baggage size can be superimposed over the detected objects so that users can visualize how close their bags are to the carry-on requirements”, see FIGs 3A-3Q).
The combination of Birchfield, Kim, and Gao then teaches identifying (225) a discrete range of acceptable values (Gao, ¶ [0044], see modified FIG. 3L above) that define a relative height of the article and a relative depth of the article, wherein the height and depth of the article are relative to the width of the article. (Birchfield, ¶ [0103]: “… In some examples, a width value, a height value, and a length value of relative dimensions represent values of a width, a height, and a length, respectively, of a 3D bounding box relative to each other”)
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 11
Regarding Claim 11, the combination of Birchfield, Kim, and Gao teaches the method of Claim 10. In addition, the combination of Birchfield, Kim, and Gao teaches the discrete range of acceptable values (Gao, ¶ [0044], see modified FIG. 3L above) that define the relative height and the relative depth of the article (Birchfield, ¶ [0103]: “… In some examples, a width value, a height value, and a length value of relative dimensions represent values of a width, a height, and a length, respectively, of a 3D bounding box relative to each other”) is determined by standard travel industry sizes for articles of baggage. (Gao, see modified FIG. 3L, each acceptable dimension value is from an airline)
Claim(s) 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Birchfield in view of Han et al. (US-20200210704-A1, hereinafter Han).
CLAIM 15
In regards to Claim 15, Birchfield teaches the method of Claim 1.
Birchfield does not explicitly disclose the 3D coordinates are calculated using a ray-casting algorithm (500) in an augmented reality environment.
Han is in the same field of art of determining dimensions/physical size of an object from image. Further, Han teaches the 3D coordinates are calculated using a ray-casting algorithm (500) in an augmented reality environment. (Han, ¶ [0038-0040]: “To determine the physical size of an object at runtime, the present disclosure may utilize both the mapping information from the marker-less AR platform 206 and the recognition result from the network-based processing system 216. The former models the environment in physical scale and the latter determines the boundary and location of the object within the environment… Using a ray cast from the camera in the direction of an object's 2D vertex within the camera view, the first plane in the point cloud hit by the ray is the object plane, and the intersection point is exactly the 3D location of the corresponding physical object vertex. Based on these vertices, the present client application 212 may derive the physical size and the model matrix M of the object”. Han teaches using an augmented reality platform to create a sparse 3D point cloud of the environment in physical scale, and an recognition system to recognize object in 2D image of the environment. Han then discloses using ray casting to obtain real-world 3D coordinate of corresponding 2D pixel of object.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Birchfield by incorporating AR platform and ray casting method that is taught by Han, to make a system to measure object’s dimensions in an augmented reality environment using ray casting algorithm; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need for a system to perform run-time object recognition and dimension determination (Han, ¶ [0030]: “The present disclosure seeks to achieve an end-to-end latency close to the inter-frame interval for continuous object recognition (e.g., without requiring a user to pause the camera at an object of interest for seconds). The present disclosure achieves this goal in part through offloading computer vision tasks to GPU resources in the edge cloud. In one example, a precise runtime object size determination capability, as well as flexible and scalable object recognition capability is integrated with a marker-less object tracking AR platform.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 16
Regarding claim 16, the combination of Birchfield and Han teaches the method of Claim 15. In addition, the combination of Birchfield and Han teaches the dimensions of the article are determined by calculating (550) the distance between 3D coordinates (Birchfield, ¶ [0088]: “A relative dimensions 126 may be a collection of data that indicates an estimation of relative dimensions (e.g., width, height, length) of a 3D bounding cuboid” Birchfield teaches dimension is determined from width, height and length of a 3D bounding cuboid of object) (Han, ¶ [0040]: “…Using a ray cast from the camera in the direction of an object's 2D vertex within the camera view, the first plane in the point cloud hit by the ray is the object plane, and the intersection point is exactly the 3D location of the corresponding physical object vertex. Based on these vertices, the present client application 212 may derive the physical size and the model matrix M of the object”) in the AR environment. (Han, ¶ [0034]: “Both ARCore and ARKit employ image frames from the monocular camera and motion data from the inertial measurement unit (IMU) to track the position and orientation of a mobile device in a 3D space”)
CLAIM 17
Regarding claim 17, the combination of Birchfield and Han teaches the method of Claim 15. In addition, the combination of Birchfield and Han teaches the ray-casting algorithm includes simultaneous localisation and mapping techniques. (Han, ¶ [0034 and 0040]: “Marker-less AR (also referred to as “dead reckoning”) may solve the simultaneous localization and mapping (SLAM) problem in run-time. Example systems include ARCore from Google, ARKit from Apple, Instant Tracking from Wikitude, and Smart Terrain from Vuforia … In one example, the marker-less AR platform 206 may generate a sparse 3D point cloud of the environment as part of the marker-less AR platform's SLAM functionality, which describes each point's location in the physical world scale”)
CLAIM 23
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Birchfield in view of Srinivasan et al. (US-11763209-B1, filed 2020, hereinafter Srinivasan), and further in view of Patel et al. (US-11006078-B1, filed 2019, hereinafter Patel).
In regards to Claim 23, Birchfield teaches the method of Claim 1.
Birchfield does not explicitly disclose the article is a bag in an airport environment and further comprising calculating a volume of carry-on baggage based on the dimensions of one or more articles associated with checked-in passengers intending to board an aircraft.
Srinivasan is in the same field of art of identifying object’s dimension from image. Further, Srinivasan teaches the article is a bag in an airport environment (Srinivasan, col 2, line 1-6: “A Virtual Measurement System for Baggage Management (“VMSBM”) uses a depth perception camera or other visual depth capturing or recording device to measure the dimensions of baggage prior to the baggage being loaded or boarded on an aircraft. Airline passengers are often allowed a carry-on allowance, which limits the size and number of carry-on bags/items that the passenger can bring into the cabin of the aircraft”) and further comprising calculating a volume of carry-on baggage based on the dimensions of one or more articles associated with checked-in passengers intending to board an aircraft. (Srinivasan, col 8, line 14-21: “Based on the estimated volume calculated from the captured dimensions and the estimated weight/volume associated with a computer bag, the system 10 estimates that the baggage weighs 15 lbs. In some embodiments, the system 10 uses this estimated weight when determining estimated fuel weights required for the flight. In other embodiments, the system 10 does not estimate weight and only stores the estimated volume of the baggage”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to substitute the target object of Birchfield’s invention with Srinivasan’s carry-on bag, to make a system to determine volume of a bag from image; thus, one of ordinary skilled in the art would be motivated to combine the references since it’s a simple substitution, Birchfield teaches determining dimensions of a 3D object and Srinivasan teaches estimating dimensions of carry-on baggage (Srinivasan, abstract: “A system that includes a depth perception camera to estimate or otherwise measure the dimensions of a proposed carry-on item to determine whether the proposed carry-on item complies with limitations associated with a user”).
The combination of Birchfield and Srinivasan does not explicitly disclose identifying the total cabin storage capacity of the aircraft, and comparing the volume of carry-on baggage with the total cabin storage capacity to identify a remaining cabin storage capacity for the aircraft.
Patel is in the same field of art of system to monitor baggage in an airport environment. Further, Patel teaches identifying the total cabin storage capacity of the aircraft (Patel, col 13, line 7-11: “the system 10 refers to the predetermined schedule to identify the total volume of available bin space, total number of average-sized carry-ons capable of being stored in the bin space, model or type of bin-space containers, etc”), and comparing the volume of carry-on baggage with the total cabin storage capacity (Patel, col 13, line 13-15: “…the size of the carry-on luggage 100 is estimated and compared to the remaining available overhead bin space…”), to identify a remaining cabin storage capacity for the aircraft. (Patel, col 13, line 15-18: “the remaining available overhead bin space includes the difference between total available overhead bin space and the used overhead bin space.”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Birchfield and Srinivasan by incorporating comparing baggage size with available storage space that is taught by Patel, to make a system to that can keep track of available carry-on storage space of an aircraft; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to avoid time delays of boarding activities (Patel, col 14, line 1-4: “As a result, the system 10 avoids time delays associated in trying to fit those bags in the overhead bins, failing to fit those bags in the overhead bins, and subsequently gate-checking those bags.”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 24
Claim(s) 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Birchfield in view of Patel et al. (US-11006078-B1, filed 2019, hereinafter Patel).
In regards to Claim 24, Birchfield teaches the method of Claim 1.
Birchfield does not explicitly disclose sending a notification if the remaining cabin storage capacity falls below a threshold value.
Patel is in the same field of art of estimate object’s dimension from image. Further, Patel teaches sending a notification if the remaining cabin storage capacity falls below a threshold value. (Patel, col 13, line 40-46: “In some embodiments, the alert that there is no more available bin space is provided to the first user. Upon receiving the alert, the carry-on luggage 100 can be checked and stored in the baggage hold instead of the overhead bin storage area in the cabin of the aircraft 80. As such, upon receiving the alert, the carry-on luggage 100 is prevented from entering the cabin of the aircraft 80”, line 55-63: “In some embodiments, the camera 25e and/or 25d transmit live video and/or images of the carry-on luggage 10 to the machine learning processing server 30, which counts the bags and detects the size of the bags. In several example embodiments, the machine learning processing server 30 determines whether the bins will be full, and/or whether the bins are close to being full, and sends an alert or message to a downline system indicating that the bins will be full and/or are close to being full”. Patel teaches sending alert if available storage is less than size of incoming baggages)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to substitute the target object of Birchfield’s invention with Patel’s carry-on bag, to make a system to determine volume of a bag from image, and use the volume information for managing airline activities; thus, one of ordinary skilled in the art would be motivated to combine the references since it’s a simple substitution, Birchfield teaches determining dimensions of a 3D object and Patel teaches estimating dimensions of carry-on baggage (Patel, col 12, last paragraph: “In some embodiments, the camera 25e is a camera that is configured to estimate dimensions and depths of an object and the camera 25e estimates the size of the carry-on luggage”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
CLAIM 25
Claim(s) 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Birchfield in view of Sheorey et al. (US-20210082149-A1, hereinafter Sheorey).
In regards to Claim 25, Birchfield teaches the method of Claim 1. In addition, Birchfield teaches a training phase to train (Birchfield, ¶ [0094-0097]: “One or more systems, such as a training framework, may train one or more neural networks of a system for object pose estimation 102 ... Training data may comprise images in which each image is associated with 2D points (e.g., vertices) of a bounding cuboid of an object depicted in the image, a centroid, and relative dimensions”) a neural network to identify the plurality of 2D pixel locations associated with the article. (Birchfield, ¶ [0081-0082]: “a system for object for pose estimation 102 (e.g., via a 2D keypoint output decoding 128) determines pixel coordinates of an input image 104 that correspond to a center of an object, and utilizes width and height values from a 2D bounding box size 114 corresponding to the pixel coordinates to determine a size of a bounding box of the object”)
Birchfield does not explicitly disclose an annotation tool (800) is used, wherein one or more of the 2D pixel locations may be identified manually (810) with the annotation tool.
Sheorey is in the same field of art of training neural network. Further, Sheorey teaches an annotation tool (800) is used, wherein one or more of the 2D pixel locations may be identified manually (810) with the annotation tool. (Sheorey, ¶ [0039]: “In addition, 2D pixel locations annotation module 303 is used to provide annotated 2D pixel locations 312 corresponding to the 3D landmarks. Such 2D pixel locations 312 may be generated using any suitable technique or techniques such as manual annotation, feature detection, or feature detection followed by manual adjustment.”)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Birchfield by incorporating annotation tool that is taught by Sheorey, to make a system to train neural network with manually annotated data; thus, one of ordinary skilled in the art would be motivated to combine the references since among its several aspects, the present invention recognizes there is a need to provide the ground-truth data for training of model (Sheorey, ¶ [0039]: “… image annotation points will be used as comparison points for projected 3D points (e.g., by differencing the positions in the normalized image coordinates in an optimization model)”).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Allowable Subject Matter
Claims 12-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The closest prior arts for Claims 12-14 are:
Birchfield et al. (US-20220277472-A1). Birchfield teaches to determine a pose and relative dimensions of an object from an image. Specifically, Birchfield disclose utilizing a perspective-n-point (PnP) algorithm to calculate a 6-DoF pose and relative dimensions of the object. Birchfield fails to disclose determining a set of 3D coordinates for each possible combination for the relative height and relative depth of the article.
Hesch et al. (Hesch, Joel A., and Stergios I. Roumeliotis. "A direct least-squares (DLS) method for PnP." IEEE, hereinafter Hesch). Hesch teaches a Direct Least-Squares (DLS) method for computing all solutions of the perspective-n point camera pose determination (PnP) problem in the general case (n ≥ 3). Birchfield fails to disclose determining a corresponding set of 2D coordinates for each set of 3D coordinates.
While both Birchfield and Hesch teach pose determination using PnP algorithm. Neither Birchfield, or Hesch, nor the combination teaches “determining a set of 3D coordinates for each possible combination for the relative height and relative depth of the article and determining a corresponding set of 2D coordinates for each set of 3D coordinates”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHUT HUY (JEREMY) PHAM whose telephone number is (703)756-5797. The examiner can normally be reached Mo - Fr. 8:30am - 6pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O'Neal Mistry can be reached on (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHUT HUY PHAM/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights