DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Application
Claim 1 was cancelled and claims 2-21 were added prior to Examination. Claims 2-21 have been examined in this application filed on or after March 16, 2013, and are being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. This communication is the First Office Action on the Merits.
Key to Interpreting this Office Action
For readability, all claim language has been bolded. Citations from prior art are provided at the end of each limitation in parenthesis. Any further explanations that were deemed necessary the by Examiner are provided at the end of each claim limitation. The Applicant is encouraged to contact the Examiner directly if there are any questions or concerns regarding the current Office Action.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 2-21 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
In regards to claims 2, 16 and 19: Applicant claims
filtering, by the data processing hardware, the first set of features to obtain a filtered set of features, wherein filtering the first set of features is based on a comparison of the sensor data to semantic data indicating one or more objects;
First, the terms “filter” and/or “filtering” is absent from Applicant disclosure in any capacity. While Applicant may be their own lexicographer, the term filtering is clearly not defined in any capacity, therefore the plain meaning of the term must be used:
Merriam-Webster dictionary defines filtering as: to remove by means of a filter.
Merriam-Webster dictionary further defines a filter in a plurality of ways, the best of which appears to be:
3: something that selectively alters or removes like a filter (as by holding back elements or modifying the appearance of something)
a: a software tool that alters the appearance of a digital image
b: software for restricting access to certain online material
Applicant disclosure citations below are best understood to be representative of the claimed filtering with regards to features, with all understood actions based on features underlined by the Examiner:
[0043] To generate the localization map 202, the semantic planner 200 is configured to confirm that a feature used as a respective localization reference point 222 corresponds to a permanent feature rather than a temporary or nonpermanent feature. In some examples, a nonpermanent feature refers to an object that undergoes some type of change in state within a period of two weeks or less. Yet in some implementations, the semantic planner 200 configures the degree of desired nonpermanence for a feature. For instance, the semantic planner 200 changes the threshold for permanence from two weeks to one week or to three weeks. In some configurations, the semantic planner 200 is part of the generation for the localization map 202 such that, during the generation of each localization reference point 222, the semantic planner 200 determines whether the localization reference point 222 corresponds to a permanent object PO or feature. Additionally or alternatively, a localization map 202 with localization reference points may be fed to the semantic planner 200 and the semantic planner 200 checks to see if one or more localization reference points should be modified or removed from the received localization map 202 because the one or more reference points correspond to a nonpermanent object NPO in the environment 10.
[0044] Referring to FIGS. 2A-2D, the semantic planner 200 includes a generator 210 and a localizer 220. The generator 210 is configured to receive sensor data 134 captured by one or more sensors 132 of the robot 100. From the sensor data 134, the generator 210 generates a plurality of localization candidates 212, 212a-n for a localization map 202 of the environment 10. Here, each localization candidate 212 corresponds to a feature or object of the environment 10 identified by the sensor data 134 and represents a potential localization reference point for the robot 100. For example, FIG. 2B depicts a view of a building environment 10 where the robot 100 is gathering sensor data 134. In this example, the generator 210 identifies five localization candidates 212, 212a-c. A first localization candidate 212, 212a corresponds to an area of a wall. A second localization candidate 212, 212b corresponds to toolboxes. A third localization candidate 212, 212c corresponds to a vertical pipe adjacent a vertical support pillar. A fourth localization candidate 212, 212d corresponds to rolls of material at a base of the vertical support pillar. A fifth localization candidate 212, 212e corresponds to a stack of cardboard boxes.
[0045] Once the generator 210 generates the plurality of localization candidates 212, the generator 210 passes the localization candidates 212 to the localizer 220. The localizer 220 is configured to determine whether the underlying feature or object corresponding to a localization candidate 212 is a permanent object PO or a nonpermanent object NPO in the environment 10. When the object corresponding to a localization candidate 212 is a permanent object PO, the localizer 220 permits or converts the localization candidate 212 to be a localization reference point 222 in the localization map 202. When the object corresponding to a localization candidate 212 is a nonpermanent object NPO, the localizer 220 prevents the localization candidate 212 from being used as a localization reference point 222 in the localization map 202.
[0048] In some implementations, to determine the permanence of an object corresponding to a localization candidate 212, the localizer 220 first determines a location for the perceived object corresponding to the localization candidate 212. In other words, the localizer 220 determines the location in the environment 10 where the sensor data 134 captured the object corresponding to the localization candidate 212. With the location of the object that relates to the localization candidate 212, the localizer 220 determines where this location occurs in the semantic model 30. In some examples, an operator 12 assists this process by indicating where a particular location in the semantic model 30 exists within the gathered sensor data 134 (or vice versa). Additionally or alternatively, the semantic planner 200 may perform a matching process that matches features from the sensor data 134 to features in the semantic model 30 in order to align the semantic model 30 and the sensor data 134. In either approach, the localizer 220 then determines whether the respective location in the semantic model 30 that matches the location of the perceived object from the sensor data 134 corresponds to a permanent object PO in the semantic model 30. Stated differently, the localizer 220 queries the semantic model 30 at a location in the semantic model 30 that corresponds to the perceived object from the sensor data 134 to determine whether semantic information 32 at that location in the semantic model 30 indicates that a permanent object PO exists in the semantic model 30 at that location. When the semantic information 32 at that location in the semantic model 30 indicates the presence of a permanent object PO, the localizer 220 enables or converts the localization candidate 212 to be a localization reference point 222 in the localization map 202.
Applicant disclosure [0045] and [0048] provide for performing comparison of the sensor data to semantic data, (i.e. a matching process, see [0048]) based on features, to determine and discriminate between permanent and non-permanent/temporary objects in an environment based on temporal status of objects (see [0046] and claim 5). Applicant further describes removal of localization reference points (i.e. - not features) based on non-permanent/temporary determination. (see [0043])
However, none of the identified components or processes are labeled or understood to be a filter, and/or perform filtering of features, as claimed, using the plain meaning of the term. There is therefore no written description support for this limitation, as claimed. Corrective action or clarification is required.
All other dependent claims of the indefinite claims detailed above are also indefinite at least by virtue of depending on the indefinite claims detailed above.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the applicant regards as the invention.
In regards to claims 2, 16 and 19: Applicant claims
filtering, by the data processing hardware, the first set of features to obtain a filtered set of features, wherein filtering the first set of features is based on a comparison of the sensor data to semantic data indicating one or more objects;
As outlined in the 35USC112(a) rejection above, there is no written description support for the filtering of features, as claimed. There is therefore also no structural support for the filtering, as claimed. The metes and bounds of the filtering is therefore unclear and indefinite. Corrective action or clarification is required.
Further in regards to claims 2, 16 and 19: During the claimed comparison, Applicant appears to be comparing sensor data to semantic data in order to discriminate between permanent and non-permanent/temporary objects in an environment based on temporal status of objects (claim 5). Applicant further describes removal of non-permanent/temporary objects from the localization map. (see [0043]) Therefore, one of ordinary skill may conclude that the claimed filtering may include removal of said non-permanent/temporary objects.
However, after the claimed filtering, claims 2, 16 and 19 claims generating, by the data processing hardware, in a first map, one or more localization reference points corresponding to the filtered set of features;
and
instructing, by the data processing hardware, performance of an action by the robot based on the location of the robot and using a second map generated based on the first set of features, wherein the first map and the second map correspond to different features.
Therefore, this interpretation is also incompatible with what is claimed because Applicant claims generating a first map corresponding to the filtered set of features, not the one or more objects indicated by semantic data.
Applicant disclosure describes generating maps 182 from received and processed sensor data 134. (see Fig. 1ZB and [0034]) Further, semantic planner 200 may generate (or modify) a localization map 202 for the robot 100, initially constructed by driving or moving the robot 100 through the environment 10 where the robot 100 will be operating and gathering sensor data 134 while the robot 100 is being driven through the environment 10. (see [0042]) To generate the localization map 202, the semantic planner 200 is configured to confirm that a feature used as a respective localization reference point 222 corresponds to a permanent feature rather than a temporary or nonpermanent feature. (see [0043]) A localization map 202 with localization reference points may be fed to the semantic planner 200 and the semantic planner 200 checks to see if one or more localization reference points should be modified or removed from the received localization map 202 because the one or more reference points correspond to a nonpermanent object NPO in the environment 10.
However, Applicant claims generating, in a first map, one or more localization reference points corresponding to the filtered set of features; understood to be localization map 202. Next, Applicant claims a second map generated based on the first set of features, wherein the first map and the second map correspond to different features. However, it is unclear which state of localization map 202 Applicant is claiming with the claimed first map and second map due to missing the necessary steps of generation of the first map prior to generating localization reference points corresponding to the filtered set of features within the first map.
Applicant then claims a second map [that is] generated based on the first set of features, (i.e. sensor data) wherein the first map and the second map correspond to different features. However, there is no prior antecedent basis for a first map with features to be different from. Further, there is no clear understanding as to whether the first map or second map contains the filtered (as best understood, removed?) objects.
All other dependent claims of the indefinite claims detailed above are also indefinite at least by virtue of depending on the indefinite claims detailed above.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 2-21, as best understood in view of 35 U.S.C. 112 rejections above, are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of parent application 17/648942, now U.S. Patent No. US 12,321,178 (herein patent ‘178). Although the claims at issue are not identical, they are not patentably distinct from each other because each and every limitation of claims 1-20 are encompassed by the limitations of claim 1-20, as follows:
Claims 2, 6, 8 and 13-14 and 19 are unpatentable over patent ‘178, claim 1.
Claims 3-4, 10 and 15 and 20 are unpatentable over patent ‘178, claim 9.
Claim 5 and 9 are unpatentable over patent ‘178, claim 4.
Claim 7 is unpatentable over patent ‘178, claim 24.
Claim 11 is unpatentable over patent ‘178, claim 11.
Claims 12 and 21 are unpatentable over patent ‘178, claim 5.
Claim 16 is unpatentable over patent ‘178, claim 12.
Claim 17 is unpatentable over patent ‘178, claim 21.
Claim 18 is unpatentable over patent ‘178, claims 12 and/or 19.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 2-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claim 2 is directed to:
2. (New) A method comprising:
receiving, by data processing hardware of a robot, sensor data from one or more sensors of the robot, the sensor data indicating a first set of features; (this is understood to be a generic data collection step. It is noted that the abstract idea outlined below does not meaningfully affect the robot or the robot sensors in any way that represents a practical application.)
filtering, by the data processing hardware, the first set of features to obtain a filtered set of features, wherein filtering the first set of features is based on a comparison of the sensor data to semantic data indicating one or more objects; (As best understood in view of Applicant disclosure, this is a data processing step that represents an abstract mental process performable by one of ordinary skill in the art mentally. This conclusion is supported by Applicant’s own disclosure [0022] “While human intelligence is naturally able to identify object permanence as a result of experience, a robot may not have the same ability.”)
generating, by the data processing hardware, in a first map, one or more localization reference points corresponding to the filtered set of features; (As best understood, this is a data processing step that represents an abstract mental process performable by one of ordinary skill in the art mentally.)
based on generating the one or more localization reference points in the first map, instructing, by the data processing hardware, determination of a location of the robot using the first map; (As best understood, this is a data processing step that represents an abstract mental process performable by one of ordinary skill in the art mentally.)
and instructing, by the data processing hardware, performance of an action by the robot based on the location of the robot (The claimed “instructing an action” is broad and generic in nature, and therefore may include a plurality of both abstract mental processes (i.e. algorithmic recognition of location, adjustment of a planned activity, etc.) as well as well-known and well-understood computer tasks grounded in processor technology (e.g. storing a variable in memory and/or generating a signal) that are not considered practical applications.)
and using a second map generated based on the first set of features, wherein the first map and the second map correspond to different features. (Similar to above, the claimed “using” is broad and generic in nature, and therefore may include a plurality of both abstract mental processes (i.e. algorithmic recognition of location, adjustment of a planned activity, etc.) as well as well-known and well-understood computer tasks grounded in processor technology (e.g. storing a variable in memory and/or generating a signal) that are not considered practical applications.)
Applying Step 1 of the Alice Analysis, the claims are understood to be directed to a process, machine, manufacture or composition of matter, and therefore we proceed to step 2A.
Applying Step 2A, Prong One of the Alice analysis, claim 2 is determined to be directed to an abstract idea (mental processes). Claim 2 is directed to data collection by a robot, processing data to determine features that meet a criteria on a map to determine a location, and generating an output based on said data processing. Claim 1 does not claim any steps that cannot be performed mentally by one of ordinary skill in the art, but is merely performed on a generic computer onboard a robot, and therefore falls within the “mental processes” grouping. See 84 Fed. Reg. 52. Because we conclude that claim 1 recites an abstract idea, we proceed to Step 2A, Prong Two.
Applying Step 2A, Prong Two of the Alice analysis, we determine whether the recited judicial exception is integrated into a practical application of that exception by: (a) identifying whether there are any additional elements recited in the claim beyond the judicial exception; and (b) evaluating those additional elements individually and in combination to determine whether they integrate the exception into a practical application. This evaluation requires an additional element or a combination of additional elements in the claim to apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the exception. If the recited judicial exception is integrated into a practical application, the claim is not “directed to” the judicial exception.
Apart from the receiving, filtering, generating and instructing steps of the abstract idea above, the only additional element recited in claim 2 is a robot comprising a sensor and processing hardware performing the abstract ideas. However, the robot is not meaningfully affected by the abstract ideas, and therefore is immaterial to the claim. Claim 2 does not recite any limitation that even generally links the use of the judicial exception to a particular technological environment. Accordingly, the language itself of claim 1 does not reflect an improvement in any particular technical field or technology. There is also no evidence that the claimed system recites an improvement to the functioning of the “computer system” itself. See MPEP § 2106.05(a). Claim 2 also does not appear to use a judicial exception in conjunction with any particular machine. See 84 Fed. Reg. 55. Accordingly, claim 2 does not integrate the judicial exception into a practical application of the exception, and we proceed to Step 2B.
Applying Step 2B of the Alice analysis, the claim(s) does/do not include additional elements beyond the judicial exception that is not “well-understood, routine, conventional” in the field or meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. The limitations are no more than a field of use or merely involve insignificant extrasolution activity. Therefore, viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself. Therefore, the claims are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter. Corrective action or clarification is required.
Independent claim 16 is the robot performing the method of claim 2, and is rejected the same or similar to claim 2, as detailed above.
Independent claim 19 is the computing system that is performing the method of claim 2, and is rejected the same or similar to claim 2, as detailed above.
Dependent claims 3-15, 17-18 and 20-21 have been evaluated in a similar manner, and do not appear to overcome these deficiencies. Therefore dependent claims 3-15, 17-18 and 20-21 are rejected in the same or a similar manner as claims 2, 16 and 19, above.
Examiner Note: Particular scrutiny is paid to dependent claim 18:
18. (New) The robot of claim 16, wherein the action comprises an action to interact with an object of the one or more objects.
Positive recitation of the performance of this action would be considered a practical application under Step 2A, Prong Two of the Alice analysis. However, it should be noted that the broadest reasonable interpretation of instruct performance of an action (per claims 1, 16 and 19) still includes mere signal generation of an instruction to interact with an object, and therefore actual performance of said interaction action is not required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 2-5, 8-11, 13-15, 19 and 21 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Kee et al. (US 20180161986 A1) herein Kee.
In regards to Claim 2, as best understood, Kee discloses the following:
2. (New) A method comprising:
receiving, by data processing hardware of a robot, sensor data from one or more sensors of the robot, the sensor data indicating a first set of features; (see at least Fig. 1 and [0025] “3D sensor assembly 120” and [0026] “A 3D range image generated by various types of camera assemblies (or combinations thereof) can be used to locate and determine the presence and location of points on the viewed object's surface” and [0033] “front end processes sensor data. It can extract and track relevant features to create the sensor measurements and can also associate the measurements with their respective objects in the factor graph.”)
filtering, by the data processing hardware, the first set of features to obtain a filtered set of features, (see at least [0039] “In the learning phase, STORM can run multiple trials in an environment to learn object classes' mobilities.”, [0041] “An object class's mobility can be represented by the covariances of its relative transformations”, [0043] “STORM can build a map of the environment while tracking objects and localizing in the map using the learned object mobilities. It can localize in a dynamic environment with static, moving, and directly manipulated objects.”, [0045] “The covariance of an object measurement can be a function of the measurement noise and the learned mobilities (covariances) of the neighboring objects. Measurements of a more static object can result in measurements with lower covariance. Accordingly, measurements with static objects can be given more weight in the factor graph optimization. Consequently, localization can depend on more static objects than less static objects, thereby improving localization accuracy and robustness in dynamic environments.”, see also [0050] “Point clouds of the background scene can exclude object points to avoid aliasing with the object point clouds. These background point clouds can be generated from the original sensor point clouds at each sensor pose. This can be done by first computing the concave hull of the objects' database point clouds in the sensor frame. Then, all points inside the hull can be removed from the background cloud.”)
wherein filtering the first set of features is based on a comparison of the sensor data to semantic data indicating one or more objects; (see at least [0005] “semantic recognition of salient objects and their positions and orientations, or poses.” and “Semantic object-based maps”, [0017] “the present invention can learn in real time the semantic properties of objects, such as the range of mobility or stasis in a certain environment (a chair moves more than a book shelf). This semantic information can be used at run time by the robot to improve its navigation and localization capabilities.”, [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility”)
generating, by the data processing hardware, in a first map, one or more localization reference points corresponding to the filtered set of features; (see at least [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility, generating a dense map of the environment, and localizing its sensor in its map relying on more static objects.” and [0020] “During the operation phase, STORM can build a map of its environment while tracking objects and localizing in the map. STORM can use the learned object mobilities to localize using more static objects.”)
based on generating the one or more localization reference points in the first map, instructing, by the data processing hardware, determination of a location of the robot using the first map; (see at least [0017] “a robot can estimate its location more robustly and accurately by relying more on static objects rather than on movable objects”, [0048] “STORM can generate the map. The robot can then use the map to navigate appropriately with respect to the environment.” and [0053] “X is defined as a sequence of multivariate random variables representing the estimated state, containing the robot pose (denoted by R.Math.X) and landmark poses”) and
instructing, by the data processing hardware, performance of an action by the robot (see at least [0022] and claim 16) based on the location of the robot and using a second map generated based on the first set of features, wherein the first map and the second map correspond to different features. (see at least [0050] “Once the factor graph is optimized, STORM can project the object models and background point clouds into a global coordinate frame. Point clouds of the background scene can exclude object points to avoid aliasing with the object point clouds. These background point clouds can be generated from the original sensor point clouds at each sensor pose. This can be done by first computing the concave hull of the objects' database point clouds in the sensor frame. Then, all points inside the hull can be removed from the background cloud.”)
In regards to Claim 3, as best understood, Kee discloses the following:
3. (New) The method of claim 2, wherein the semantic data further indicates a second set of features corresponding to the one or more objects, (see at least [0005] “semantic recognition of salient objects and their positions and orientations, or poses.” and “Semantic object-based maps”, [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility”) and wherein filtering the first set of features comprises filtering a feature from the first set of features based on determining that the feature does not correspond to a feature of the second set of features. (see at least [0050] “Point clouds of the background scene can exclude object points to avoid aliasing with the object point clouds. These background point clouds can be generated from the original sensor point clouds at each sensor pose. This can be done by first computing the concave hull of the objects' database point clouds in the sensor frame. Then, all points inside the hull can be removed from the background cloud.”)
In regards to Claim 4, as best understood, Kee discloses the following:
4. (New) The method of claim 2, wherein the semantic data further indicates a second set of features corresponding to the one or more objects, (see at least [0005] “semantic recognition of salient objects and their positions and orientations, or poses.” and “Semantic object-based maps”, [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility”) wherein the semantic data comprises a semantic model of an environment of the robot, (see at least [0007] “STORM models the trajectories of objects rather than assuming the objects remain static.”, [0021] and [0031] “environment model” and [0049]-[0050] “object models”) and wherein filtering the first set of features comprises:
identifying a location within the environment; (see at least [0017] “a robot can estimate its location more robustly and accurately by relying more on static objects rather than on movable objects”, [0048] “STORM can generate the map. The robot can then use the map to navigate appropriately with respect to the environment.” and [0053] “X is defined as a sequence of multivariate random variables representing the estimated state, containing the robot pose (denoted by R.Math.X) and landmark poses”)
determining that a feature of the first set of features corresponds to the location within the environment based on the sensor data; (see at least above citations to [0017])
determining that a location in the semantic model corresponding to the location within the environment does not correspond to a feature of the second set of features; (see at least [0043] “[0043] During the operation phase, STORM can build a map of the environment while tracking objects and localizing in the map using the learned object mobilities. It can localize in a dynamic environment with static, moving, and directly manipulated objects.”)
and
filtering the feature from the first set of features based on determining that the feature corresponds to the location within the environment and determining that the location in the semantic model does not correspond to a feature of the second set of features. (see at least [0050] “Once the factor graph is optimized, STORM can project the object models and background point clouds into a global coordinate frame. Point clouds of the background scene can exclude object points to avoid aliasing with the object point clouds. These background point clouds can be generated from the original sensor point clouds at each sensor pose. This can be done by first computing the concave hull of the objects' database point clouds in the sensor frame. Then, all points inside the hull can be removed from the background cloud.”)
In regards to Claim 5, as best understood, Kee discloses the following:
5. (New) The method of claim 2, wherein the semantic data indicates a temporal status of an object of the one or more objects, wherein a feature of the first set of features corresponds to the object, and wherein filtering the first set of features comprises filtering the feature from the first set of features based on the temporal status of the object. (see at least [0019] “These measurements are a potential technique by which STORM to determine the object class's mobility metric. The mobility metric can be a measure of how mobile or static an object or class of objects are, after a number of observations over a certain time window.” and [0045] “[0045] The covariance of an object measurement can be a function of the measurement noise and the learned mobilities (covariances) of the neighboring objects. Measurements of a more static object can result in measurements with lower covariance. Accordingly, measurements with static objects can be given more weight in the factor graph optimization. Consequently, localization can depend on more static objects than less static objects, thereby improving localization accuracy and robustness in dynamic environments.”)
In regards to Claim 8, as best understood, Kee discloses the following:
8. (New) The method of claim 2, wherein instructing the determination of the location of the robot comprises instructing the determination of the location of the robot relative to a localization reference point of the one or more localization reference points. (see at least previous citations, see also Fig. 2 and [0030] “robot poses” and [0053] “X is defined as a sequence of multivariate random variables representing the estimated state, containing the robot pose (denoted by R.Math.X) and landmark poses”)
In regards to Claim 9, as best understood, Kee discloses the following:
9. (New) The method of claim 2, wherein the semantic data further indicates a time period associated with the one or more objects, and wherein filtering the first set of features is further based on the time period. (see at least [0019] “These measurements are a potential technique by which STORM to determine the object class's mobility metric. The mobility metric can be a measure of how mobile or static an object or class of objects are, after a number of observations over a certain time window.”)
In regards to Claim 10, as best understood, Kee discloses the following:
10. (New) The method of claim 2, further comprising:
aligning the sensor data and the semantic data, wherein filtering the first set of features is further based on aligning the sensor data and the semantic data. (see at least Fig. 2 and [0030] “robot poses” and [0053] “X is defined as a sequence of multivariate random variables representing the estimated state, containing the robot pose (denoted by R.Math.X) and landmark poses”, see also [0035]-[0036] “PoseNet”)
In regards to Claim 11, as best understood, Kee discloses the following:
11. (New) The method of claim 2, wherein the semantic data comprises a three- dimensional representation of an environment of the robot. (see at least [0024]-[0026] “3D image data 124 in the form of range images and/or point 3D point clouds” and [0061] “multitude of 3D processing algorithms”)
In regards to Claim 13, as best understood, Kee discloses the following:
13. (New) The method of claim 2, wherein the semantic data further indicates a second set of features corresponding to the one or more objects, (see at least [0005] “semantic recognition of salient objects and their positions and orientations, or poses.” and “Semantic object-based maps”, [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility”) and wherein instructing performance of the action by the robot is further based on the second set of features. (see at least [0022] and claim 16)
In regards to Claim 14, as best understood, Kee discloses the following:
14. (New) The method of claim 2, wherein the semantic data further indicates an obstruction or a mobility associated with one or more features of the first set of features. [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility”)
In regards to Claim 15, as best understood, Kee discloses the following:
15. (New) The method of claim 2, wherein the semantic data further indicates a second set of features corresponding to the one or more objects, (see at least [0005] “semantic recognition of salient objects and their positions and orientations, or poses.” and “Semantic object-based maps”, [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility”) the method further comprising:
comparing the first set of features to the second set of features, wherein the comparison of the sensor data to the semantic data is based on comparing the first set of features to the second set of features. (see at least [0019] “These measurements are a potential technique by which STORM to determine the object class's mobility metric. The mobility metric can be a measure of how mobile or static an object or class of objects are, after a number of observations over a certain time window.” see also Fig. 2 and [0030] “robot poses” and [0053] “X is defined as a sequence of multivariate random variables representing the estimated state, containing the robot pose (denoted by R.Math.X) and landmark poses”, see also [0035]-[0036] “PoseNet”)
In regards to Claim 19: Claim 19 is the computing system performing the method of claim 1. Kee discloses a computing system (see Fig. 1, item 116) and the method of claim 1, therefore claim 19 is rejected the same or similar to claim 1, above.
In regards to Claim 21, as best understood, Kee discloses the following:
21. (New) The computing system of claim 19, wherein the one or more objects correspond to at least one of a wall, a door, a window, a fixture, or equipment within an environment of the robot. (see at least [0027] “at least one of the objects defines a relatively fixed or stationary object 152 (e.g. a window, shelf, etc.)”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Kee et al. (US 20180161986 A1), herein Kee, in view of Hyung et al. (US 20100161225 A1), herein Hyung.
In regards to claim 6, as best understood, Kee does not explicitly disclose the following, which is taught by Hyung:
6. (New) The method of claim 2, wherein the second map indicates a no-step region or an obstacle corresponding to a feature of the first set of features. (see at least Fig. 1 and [0043] “movable robot body 12 and an image acquisition unit 14 mounted to the robot body 12, moves self-controllably in an unknown dynamic environment.”, [0047] “obstacle avoidance path through the chase of dynamic objects.”, [0076] “when it is determined at operation 110 that the feature points are not included in the static cluster, the control unit 18 registers the feature points in a dynamic map (120), and extracts the average velocity of the dynamic clusters registered in the dynamic map (122) to pre-estimate a path of the dynamic clusters (124). Subsequently, the control unit 18 pre-estimates a path along which the dynamic objects will move to create a dynamic object avoidance path (126).”)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the features of Hyung with the invention of Kee, with a reasonable expectation of success, with the motivation of providing localization technology for mobile products usable indoors or in downtown areas where global positioning system (GPS) does not exhibit excellent performance. (Hyung, [0005])
In regards to claim 7, as best understood, Kee discloses the following:
7. (New) The method of claim 2, wherein filtering the first set of features comprises filtering a feature from the first set of features, (see at least previous citations)
Kee does not explicitly disclose the following, which is taught by Hyung:
and wherein the second map indicates a no-step region or an obstacle corresponding to the feature. (see at least Fig. 1 and [0043] “movable robot body 12 and an image acquisition unit 14 mounted to the robot body 12, moves self-controllably in an unknown dynamic environment.”, [0047] “obstacle avoidance path through the chase of dynamic objects.”, [0076] “when it is determined at operation 110 that the feature points are not included in the static cluster, the control unit 18 registers the feature points in a dynamic map (120), and extracts the average velocity of the dynamic clusters registered in the dynamic map (122) to pre-estimate a path of the dynamic clusters (124). Subsequently, the control unit 18 pre-estimates a path along which the dynamic objects will move to create a dynamic object avoidance path (126).”)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the features of Hyung with the invention of Kee, with a reasonable expectation of success, with the motivation of providing localization technology for mobile products usable indoors or in downtown areas where global positioning system (GPS) does not exhibit excellent performance. (Hyung, [0005])
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Kee et al. (US 20180161986 A1), herein Kee, in view of Aghamohammadi et al. (US 20100161225 A1), herein Aghamohammadi.
In regards to claim 12, as best understood, Kee discloses the following:
12. (New) The method of claim 2, further comprising:
identifying an object of the one or more objects based on the semantic data; (see at least [0061] “ConvNet architecture of SegNet can be trained on data containing various objects that are trained and stored in an appropriate database. SegNet segments the image and passes the mask to a program which crops the point cloud from the sensor at block 506. SegNet then outputs a segmented mask with pixel wise semantic object labels.”)
Kee does not explicitly disclose the following, which is taught by Aghamohammadi:
and instructing the one or more sensors to capture at least a portion of the sensor data in response to identifying the object. (see at least [0063] “When a candidate object is detected, object recognition techniques may be used to identify the candidate object.”, [0064] “If the candidate object is the object of interest… object localization may be performed to determine the location of the object or part of the object in the image. In some aspects, a bounding box may be formed around the object. In doing so, the scale and location of the object may be determined. Based on this information and the location of the camera, control input may be determined to move the camera to better center the object within the bounding box.”)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the features of Aghamohammadi with the invention of Kee, with a reasonable expectation of success, with the motivation of providing autonomous systems, such as robots, the ability to make decisions in view of uncertainty regarding the location and identity of certain objects within the environment. (Aghamohammadi, [0005])
Claims 16-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kee et al. (US 20180161986 A1), herein Kee, in view of Perkins et al. (US 20200302207 A1), herein Perkins.
In regards to claim 16, as best understood, Kee discloses the following:
16. (New) A robot (see at least Fig. 1 and [0022] “moving robot 110”) comprising:
a body; (see at least Fig. 1)
Kee does not explicitly disclose the following, which is taught by Perkins:
two or more legs coupled to the body; (see at least [0029] “The legs 120 are locomotion-based structures (e.g., legs and/or wheels) that are configured to move the robot 100 about the work environment 10. The robot 100 may have any number of legs 120 (e.g., a quadruped with four legs, a biped with two legs, a hexapod with six legs, an arachnid-like robot with eight legs, etc.).”)
As best understood, Kee discloses the following:
one or more sensors coupled to the body; (see at least Fig. 1 and [0025] “3D sensor assembly 120” and [0026] “A 3D range image generated by various types of camera assemblies (or combinations thereof) can be used to locate and determine the presence and location of points on the viewed object's surface”
data processing hardware; and memory hardware in communication with the data processing hardware, (see Fig. 1, item 116 and [0023])
the memory hardware storing instructions, wherein, based on execution of the instructions, the data processing hardware is configured to:
receive sensor data from the one or more sensors, the sensor data indicating a first set of features; (see previous citations to claim 2)
filter the first set of features to obtain a filtered set of features, wherein filtering the first set of features is based on a comparison of the sensor data to semantic data indicating one or more objects; (see previous citations to claim 2)
generate, in a first map, one or more localization reference points corresponding to the filtered set of features; (see previous citations to claim 2)
based on generating the one or more localization reference points in the first map, instruct determination of a location of the robot using the first map; (see previous citations to claim 2)
and instruct performance of an action by the robot based on the location of the robot and using a second map generated based on the first set of features, wherein the first map and the second map correspond to different features. (see previous citations to claim 2)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the features of Perkins with the invention of Kee, with a reasonable expectation of success, with the motivation of building an accurate map of surroundings and localizing itself in the map for a robot to enable intelligent operation in an unknown environment using sensor data, (Kee, [0002]) and/or with the motivation of providing robot mobility that can be utilized in a variety of industries including, for example, hazardous environments and exploration. One of ordinary skill would clearly understand that a robot with legs may be capable of exploring and performing tasks in hazardous areas where a wheeled robot cannot go.
In regards to claim 17, as best understood, Kee does not explicitly discloses the following, which is better taught by Perkins:
17. (New) The robot of claim 16, wherein the robot further comprises an arm coupled to the body, (see at least [0037] “robot 100 further includes one or more appendages, such as an articulated arm 150 (also referred to as an arm or a manipulator arm) disposed on the body 110”) and wherein the two or more legs comprise four legs. (see at least [0029] “The legs 120 are locomotion-based structures (e.g., legs and/or wheels) that are configured to move the robot 100 about the work environment 10. The robot 100 may have any number of legs 120 (e.g., a quadruped with four legs, a biped with two legs, a hexapod with six legs, an arachnid-like robot with eight legs, etc.).”)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the features of Perkins with the invention of Kee, with a reasonable expectation of success, with the motivation of building an accurate map of surroundings and localizing itself in the map for a robot to enable intelligent operation in an unknown environment using sensor data, (Kee, [0002]) and/or with the motivation of providing robot mobility that can be utilized in a variety of industries including, for example, hazardous environments and exploration. One of ordinary skill would clearly understand that a robot with arms and legs may be capable of exploring and performing tasks in hazardous areas where a wheeled robot cannot.
In regards to claim 18, as best understood, Kee discloses the following:
18. (New) The robot of claim 16, wherein the action comprises an action to interact with an object of the one or more objects. (see at least [0007] “This solution enables a more flexible way for robots to interact with and manipulate an unknown environment... STORM allows enhanced freedom in manipulation of objects,” [0037] “an integrated strategy for planning, perception, state-estimation, and action in complex mobile manipulation domains can be based on planning” and “manipulation goals” and [0046] “When an object is directly manipulated…”)
In regards to claim 20, as best understood, Kee does not explicitly discloses the following, which is better taught by Perkins:
20. (New) The computing system of claim 19, wherein the first set of features corresponds to a set of objects located in an environment of the robot, (see at least [0022] “the work environment 10 includes a plurality of boxes 20, 20a-n stacked on a pallet 30 lying on a ground surface 12.” and [0047] “image processing system 200 is configured to detect shapes corresponding to one or more boxes 20 within the work environment 10 about the robot 100… In some examples, the robot 100 detects one or more box 20 and communicates a location of the box 20 to another entity (e.g., a worker, another robot, an owner of the box 20, etc.)… the robot 100 may be aware of constraints such as a strength of the robot 100 or a size of a box 20 that the robot 100 is able to manipulate.” and [0060] “image processing system 200 utilizes the monocular image 176a for several processing steps to generate geometric features of the target box 202 (e.g., determining corners 214, detecting edges 228, estimating faces 224, etc.)”)
As best understood, Kee discloses the following:
wherein the one or more objects are located in the environment, and wherein the semantic data further indicates a second set of features corresponding to the one or more objects. (see at least [0005] “semantic recognition of salient objects and their positions and orientations, or poses.” and “Semantic object-based maps”, [0018] “STORM can simultaneously execute a number of operations: identifying and tracking objects (static, moving, and manipulated) in the scene, learning each object class's mobility”)
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to combine the features of Perkins with the invention of Kee, with a reasonable expectation of success, with the motivation of building an accurate map of surroundings and localizing itself in the map for a robot to enable intelligent operation in an unknown environment using sensor data, (Kee, [0002]) and/or with the motivation of providing robot mobility that can be utilized in a variety of industries including, for example, hazardous environments and exploration. One of ordinary skill would clearly understand that a robot with arms and legs may be capable of exploring and performing tasks in hazardous areas where a wheeled robot cannot.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jason Roberson, whose telephone number is (571) 272-7793. The examiner can normally be reached from Monday thru Friday between 8:00 AM and 4:30 PM. The examiner may also be reached through e-mail at Jason.Roberson@USPTO.GOV, or via FAX at (571) 273-7793. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Navid Z Mehdizadeh can be reached on (571)-272-7691.
Another resource that is available to applicants is the Patient Application Information Retrieval (PAIR) system. Information regarding the status of an application can be obtained from the PAIR system. Status information for published applications may be obtained from either Private PAIR or Public PAX. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have any questions on access to the Private PAIR system, please feel free to contact the Electronic Business Center (EBC) at 866-217-9197 (toll free).
Applicants are invited to contact the Office to schedule either an in-person or a telephone interview to discuss and resolve the issues set forth in this Office Action. Although an interview is not required, the Office believes that an interview can be of use to resolve any issues related to a patent application in an efficient and prompt manner.
Sincerely,
/JASON R ROBERSON/
Patent Examiner, Art Unit 3669
June 19, 2026
/NAVID Z. MEHDIZADEH/Supervisory Patent Examiner, Art Unit 3669
Prosecution Insights