DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
General Note
Typographical errors with respect to the claim numbering of 35 USC § 103 rejections have been corrected. While the detailed rejections provided were correct and were of Claims 4 – 5, listing of claim statements were made with respect to Claims 5 – 6 and their corresponding equivalent claims (12 – 13 and 19 – 20). This error has been corrected to refer to Claims 4 – 5 and their corresponding equivalent claims. Examiner notes the correct claims and rejections made were self-evident from the detailed rejections made in the Office Action.
Claim Objections
The objections to the claims are withdrawn in light of Applicant’s amendments correcting the issues.
Claim Rejections - 35 USC § 112(b)
The rejections of the claims are withdrawn in light of Applicant’s amendments correcting the issues.
Claim Rejections - 35 USC § 102
Applicant's arguments filed 07/07/2026 have been fully considered but they are not persuasive.
Applicant’s arguments are replete with issues which may generally be addressed under the following categories:
(1) Applicant’s arguments appear to rely on interpretations of the claim language much narrower than the broadest reasonable interpretation thereof. Applicant’s terms comprise well understood general terms of phrase for the art and thus have an ordinary and customary meaning to those of ordinary skill in the art, enough if such meaning is derived simply from the dictionary definition(s) of the term(s) comprising a particular noun phrase. Furthermore, there does not appear to be any express intent in the specification to provide a special definition for any of the terms in the claims. Applicant’s arguments are furthermore silent with respect to any such special definitions. MPEP 2111.01 which discusses the “plain meaning” of the words of claims relates.
Examiner notes e.g. that the term “industrial” which is relied upon with respect to various arguments was discussed in paragraph 10. of the Claim Interpretation section of the Office Action dated 04/07/2026. Applicant does not appear to have addressed this or any other claim term interpretation presented therein and repeated and/or updated within this Office Action.
(2) Related to (1), but also distinct therefrom, applicant's arguments frequently indicate that the reference(s) fail to show certain features of the invention. However, in all instances Examiner found that the relied upon feature was not actually recited in the rejected claim(s), at a minimum at the level of detail and/or narrowness argued.
Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
If Applicant believes these features to distinguish Applicant’s originally filed disclosure from the prior art, Applicant is invited to amend the claims such that these features are actually recited within the claims, or recited at the level of specificity/narrowness argued.
(3) Related to (1), but provided alternatively to (1), Applicant’s arguments rely on language solely recited in preamble recitations. When reading the preamble in the context of the entire claim, the recitation “for industrial part sorting processing” is not limiting because the body of the claim describes a complete invention and the language recited solely in the preamble does not provide any distinct definition of any of the claimed invention’s limitations. Thus, the preamble of the claim(s) is not considered a limitation and is of no significance to claim construction. See Pitney Bowes, Inc. v. Hewlett-Packard Co., 182 F.3d 1298, 1305, 51 USPQ2d 1161, 1165 (Fed. Cir. 1999). See MPEP § 2111.02(II) relates. See e.g.:
“If the body of a claim fully and intrinsically sets forth all of the limitations of the claimed invention, and the preamble merely states, for example, the purpose or intended use of the invention, rather than any distinct definition of any of the claimed invention’s limitations, then the preamble is not considered a limitation and is of no significance to claim construction. Shoes by Firebug LLC v. Stride Rite Children’s Grp., LLC, 962 F.3d 1362, 2020 USPQ2d 10701 (Fed. Cir. 2020)”
Furthermore, Examiner notes that the term “industrial” was discussed in paragraph 10. of the Claim Interpretation section of the Office Action dated 04/07/2026, and thus regardless of the above Applicant’s arguments related to the term appear moot. Applicant has not addressed this or any other claim term interpretation presented therein.
(4) A recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim. See again the Claim Interpretation section of the Office Action dated 04/07/2026. Applicant does not appear to have addressed this or any other claim term interpretation presented therein and repeated and/or updated within this Office Action, which included discussion of terms interpretable as indicating an intended use, result, or similar.
Examiner additionally notes, and which is related to all of the items above, that Applicant provides broad and lengthy recitations of limitations (features) which Applicant indicates that “Huang does not disclose or suggest”. The arguments following refer to these limitations as a) through d), and often do so without referring clearly to the particular portion of the limitation(s) in question. Furthermore, this lack of particular reference and/or recitation makes clear in many instances that the asserted undisclosed feature is likewise un-claimed by nature of the critical portions of the feature not being quoted by Applicant ((2) above relates). Paraphrasing, while potentially useful in explaining Applicant’s intended or expected interpretation of a claim, does not clearly indicate what limitation in particular Applicant believes is different.
For example, Applicant frequently uses the term “fine-tuned” with respect to terms which are only recited as “pre-trained”. The terms “fine-tuned”’ and “pre-trained” are distinctly different. It is unclear if Applicant believes there to be additional features of fine tuning, or if Applicant believes these terms to be equivalent, etc., and furthermore still what the distinction in question even is from the prior art.
The following will be a concise attempt to address at least the majority of the arguments made, such that at least all of the arguments are categorically addressed, using reference to the above categories where appropriate for the sake of brevity:
With respect to (a) Applicant argues “the input is structurally defined to contain two semantic components” followed by even more specific definitions, that it cannot be “an unparsed, monolithic sentence”, that it must “be decomposed into, or presented as, a separate "feature description" and a separate "sorting requirement description,"”. No such limitation is found in the claims. See (2) above. Furthermore, the term/noun phrase and subsidiary terms/noun phrases of “comprises a feature description and a sorting requirement description of to-be-sorted parts” does not require such a narrow interpretation. See (1) above.
Applicant appears to argue another feature, that of “a task-instruction-understanding model … fine-tuned for industrial part-sorting” which “generate a sorting control signal based on those parsed fields”. Again, this is not claimed. See (1) and (2) above. Applicant does not define the term “industrial” in any manner within the claims or the specification.
Applicant appears to argue another feature of “industrial part attributes (e.g., material "stainless steel", shape "hex head", use "bearing") distinct from the household "trash/blue tray" example of Huang”. Again, this is not claimed. See (1), (2), and (3) above.
With respect to (b) Applicant argues “claim 1 requires that the task instruction understanding model be fine-tuned for an industrial part-sorting task”. This is not claimed. See (1) and (2) above.
Applicant appears to argue another feature of “that [the task instruction understanding model] extracts the sorting requirement from the structured part-sorting description information and generates the sorting control signal”. This is not claimed. See (1) and (2) above. Examiner notes that “by adopting a task instruction understanding model” is not specific in any way, particularly in the specific manner claimed.
Applicant appears to argue another feature of “(i) fine-tuning the LLM on labeled part-sorting instruction-control pairs”. This is not claimed. See (2) above. The words “label”, “pair”, their equivalents, or even anything close to this limitation is not present in the claims.
Applicant appears to argue another feature of “generating a sorting control signal”. Huang very clearly discloses generation of a control signal for sorting. This furthermore appears to potentially relate to how this feature is claimed. See (1), (2), and (4) above.
With respect to (c) Applicant argues a feature of a “unified "intelligent perception model for parts" that (a) takes RGB-D images as input, (b) contains a Transformer layer, a CNN layer and a fully-connected layer (see dependent claims)”. The term “unified” is absent from the claims”. Furthermore, limitiations of dependent claims are only relevant to the patentability of the claims in which they are recited, and thus such a blanket statement is inapplicable to the independent claims at a minimum. Furthermore, these features, if even claimed, appear to be with respect to dependent claims not rejected under Huang alone, but Huang in combination with other references under 35 USC 103, and thus these arguments are irrelevant. See (1) and (2) above.
Applicant appears to argue another feature of a “limitation that the perception model is a single pre-trained neural network model which determines (i.e., outputs) both the category and the grasp pose based on image recognition analysis of the to-be-sorted part”. The claim does not recite “single” or equivalent anywhere in the claims.
Examiner notes that the term “model” was discussed in paragraph 9. of the Claim Interpretation section of the Office Action dated 04/07/2026. Applicant does not appear to have addressed this or any other claim term interpretation presented therein and repeated and/or updated within this Office Action. This interpretation appears critical to Applicant’s arguments. Thus, (1) and (2) above relate.
With respect to (d) Applicant argues a definition of the term “intelligent obstacle-avoidance neural network model”, stating “i.e., a trained neural network that processes sensory data -e.g., voxel data-and outputs an obstacle representation or directly influences trajectory generation”. This definition of the term is not claimed nor presented as a special definition in the originally filed disclosure. See (1) and (2) above.
Applicant furthermore states in relation to this that, “Huang contains no neural network”. This is categorically incorrect. Huang recites various “neural network” (See e.g. LLMs, VLMs, and others).
Applicant furthermore states in relation to this that, “let alone a dedicated obstacle-avoidance neural network-for this purpose”. Note the lack of the phrase “dedicated”, the implied distinctness and separateness of the phrasing, and “for this purpose”. None of this is claimed, or in the case of “purpose” relevant. See again (1), (2), and (4) above.
Claim Rejections - 35 USC § 103
Applicant's arguments filed 07/07/2026 have been fully considered but they are not persuasive.
First, Examiner notes that all of Applicant’s arguments appear to argue the references individually. One cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Any arguments against a rejection made under 35 USC § 103 must address the combination of references, in particular and at least any combination contemplated by the Office Action.
Consequently, all of Applicant’s arguments are considered unpersuasive based on this fact alone.
Second, none of the limitations addressed by Applicant are made with respect to any of the claim limitations actually rejected under 35 USC § 103. The limitations of (a) – (d) are only relevant with respect to any curing of potentially deficiencies that a potential 103 rejection of these limitations may encompass and thus these are a continuation of arguments for patentability with respect to the claims rejected under 35 USC § 102, not 103.
Applicant does not appear to address any of the rejections made under 35 USC § 103. At the very least, Applicant does not clearly refer to any claim limitation specifically rejected under any of the rejections made under 35 USC § 103, and only appears to refer back to (a) – (d).
Consequently, all of Applicant’s arguments are considered unpersuasive based on this fact alone.
Third and finally, the arguments appear to exhibit the same issues as categorically addressed above with respect to (a) – (d) under (1) – (4).
As a specific example, Applicant argues “processing RGB and depth images as dual inputs”. However, the claims explicitly recite “an image of a to-be-sorted part” wherein “the image comprises an RGB image and a depth image”. The claim does not provide for these to be separate and distinct images or similarly, separate and distinct inputs to a model. Additionally, Huang discloses the use of up to four RGB-D cameras simultaneously (Appendix A.5), thus reading on the limitation in a variety of ways as presently constructed. For example, the claim does not limit the “industrial camera” itself in any manner such as the quantity of cameras involved. As appears potentially argued, there would inherently need to be at least two cameras that could be considered separate and distinct. Additionally, an RGB-D camera as disclosed in Huang will take output either an RGB and D image, or an RGB-D (combined) image, and will do so using two separate cameras/sensors. Furthermore, an RGB-D image comprises both “an RGB image and a depth image” even if the manner in which it is achieved is through simultaneous/combined R, G, B, and D image channels. Examiner notes that Applicant’s disclosure appears wholly silent with respect to the particular structure of the camera in question. In summary, Applicant has not shown a required and evident technical distinction in the present claim language.
Consequently, all of Applicant’s arguments are considered unpersuasive based on these issues alone.
Claim Interpretation
General Notes:
Applicant makes frequent use of terms which under their plain meaning would appear to not have any particular distinction without further limitation being provided within the claims, and furthermore leave the claims as potentially disconnected and broader as they do not refer back to items which may normally as disclosed be the same. Examples include:
“feature description” introduced as part of what comprises the “part sorting description information” and “feature … of the to-be-sorted parts”.
“part sorting description information”, “sorting requirement description”, and “sorting requirement”
“feature description” and “feature”.
Applicant has made frequent use of phrases such as “for” and “to”. In some cases, these terms may not amount to an actual positive recitation of a claim limitation and appear to only be provided for context of the actual limitations within the claim or otherwise indicate an intended result, use, or purpose of a preceding limitation. While in the interest of compact prosecution and where expedient Examiner has provided prior art which Examiner believes teaches these recitations, whether considered positively recited or not, Applicant is advised to review their claims and validate all claim limitations which Applicant desires to be clearly and positively recited. This may require amendment to more clear and precise phrasing or construction.
For example, Claim 1 recites “performing … analysis processing … to obtain …”. If the act of obtaining is desired to be very clearly and explicitly claimed it would better be constructed as “obtaining … by performing … analysis processing” or similar, wherein the action is claimed, and the exact means by which the action is performed, accomplished, etc. is further recited in a clearly linked manner (e.g. “by”).
Applicant recites various “model”. However, without further limitation to these terms, there is no inherent requirement that a given model is distinct, independent, and/or separate from another. A computer implemented model, unless otherwise specified by a claim, is understood to be distinguished by the function indicated by the words modifying the word “model”. Function and structure are distinct, and thus this is effectively a matter of arbitrary labelling without further limitation being provided. A given model may perform one or many different functions, and may comprise one or many different functions, and may even itself comprise one or many models.
The adjective “industrial” is frequently used before other terms. This does not appear to inherently alter the scope of the terms without further limitations being provided, which appear absent in the present claims, the term “industrial” being particularly broad and inherent under the broadest reasonable interpretation thereof through mere context of a “robotic arm” already required in the limitations of all of the claims.
The term “feature description” and “sorting requirement” have especially broad meaning under their plain English meaning. Furthermore, even as disclosed these terms appear particularly broad. See [0052] – [0054] of Applicant’s specification which recites:
“The part sorting description information includes the feature description and the sorting requirement description of the to-be-sorted parts, which are formed by natural language descriptions. For example, the part description information is “sort all bearing parts into one box”.
The feature description mainly defines an attribute of the to-be-sorted parts, such as the material, use and shape of the to-be-sorted parts.
The sorting requirement description mainly refers to an instruction of where the parts are sorted, such as how many boxes the to-be-sorted parts are to be sorted into”
The disclosures use of “such as” indicates that these are mere examples and non-limiting.
The terms “global vision feature” and “local global feature” do not appear to be disclosed with any particularity (and consequently also not claimed with any particularity). They thus appear to be especially broad terms.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 – 3, 6, 8 – 10, 13, 15 – 17, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Huang et al. (Huang, Wenlong, et al. "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.05973 (2023)).
Regarding Claim 1, Huang teaches:
An embodied intelligence-based method for industrial part sorting processing, comprising:
receiving part sorting description information inputted by a user, wherein the part sorting description information comprises a feature description and a sorting requirement description of to-be-sorted parts (See at least Page 1, Abstract “In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects” and Figure 1 wherein the first image-paired example of a user instruction is “Sort trash to blue tray”. Feature descriptions include “trash”, “blue”, and “tray”. Sorting requirement descriptions include “trash”, “blue”, and “tray”. See various other examples throughout, such as those in Figure 3 as necessary);
performing, according to the part sorting description information, analysis processing by adopting a task instruction understanding model (See at least Large Language Model in Figure 2), to obtain a control signal for controlling a robotic arm to sort the to-be-sorted parts (See at least Figure 3 illustrating controlled robot actions), wherein the task instruction understanding model is a neural network model which is pre-trained based on a large language model (See at least Page 2, Section 2, “In this work, we leverage LLMs for zero-shot in-the-wild cost specification with superior generalization” and Page 3, Section 2, “we rely on pre-trained language models for their open-world knowledge and tackle the more challenging robotic manipulation in 3D”) for extracting a sorting requirement from text and generating a sorting control signal based on the sorting requirement (See Figure 2 wherein user input is taken which may include as shown in Figure 1 sorting requirements in text and generates code);
performing, according to the control signal, analysis processing on an image of a to-be-sorted part by adopting an intelligent perception model for parts (See at least Page 4, Section 3.2, “Specifically, when an instruction is given as a comment in the code, LLMs can be prompted to 1) call perception APIs (which invoke vision language models (VLM) such as an open-vocabulary detector [13–15]) to obtain spatial-geometrical information of relevant objects, 2) generate NumPy operations to manipulate 3D arrays, and 3) pre scribe precise values at relevant locations”), to obtain a category and a grasp pose of the to-be-sorted part (See at least Pages 3 and 4, Section 3.1, “The central problem investigated in this work is to generate a motion trajectory τri for robot r and each manipulation phase described by instruction ℓi. We represent τri as a sequence of dense end-effector waypoints to be executed by an Operational Space Controller [117], where each waypoint consists of a desired 6-DoF end-effector pose, end-effector velocity, and gripper action”), wherein the image is collected by an industrial camera (See at least Page 20, Section A.4, “We mount two RGB-D cameras (Azure Kinect) at two opposite ends of the table: bottom right and top left from the top down view. At the start of each rollout, both cameras start recording and return the real-time RGB-D observations at 20 Hz” and Figure 2, in particular Cam #1 and Cam #2), and the intelligent perception model for parts is a neural network model which is pre-trained for determining a category and a grasp pose of a part based on image recognition analysis (See again at least Page 4, Section 3.2 with respect to the Vision Language Model);
controlling the robotic arm to sort the to-be-sorted part according to the control signal, the category and the grasp pose of the to-be-sorted part (See at least Pages 3 and 4, Section 3.1, “The central problem investigated in this work is to generate a motion trajectory τri for robot r and each manipulation phase described by instruction ℓi. We represent τri as a sequence of dense end-effector waypoints to be executed by an Operational Space Controller [117], where each waypoint consists of a desired 6-DoF end-effector pose, end-effector velocity, and gripper action” and Figure 3 illustrating control for several tasks), and performing obstacle-avoidance processing through an intelligent obstacle-avoidance neural network model (See at least Page 1, Abstract, “We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations” and Page 6, Section 4, “Motion Planner. We consider only affordance and avoidance maps in the planner optimization, which finds a sequence of collision-free end-effector positions p1:N ∈ R3 using greedy search”) in a sorting process (See above context).
Regarding Claim 2, Huang teaches:
The method according to claim 1, wherein performing the obstacle-avoidance processing through the intelligent obstacle-avoidance neural network model in the sorting process comprises:
in the sorting process, processing, by adopting the intelligent obstacle-avoidance neural network model, voxel data of different angles of physical environment around the robotic arm which is collected in real time by the industrial camera (See at least Page 20, Section A.4, “We mount two RGB-D cameras (Azure Kinect) at two opposite ends of the table: bottom right and top left from the top down view. At the start of each rollout, both cameras start recording and return the real-time RGB-D observations at 20 Hz” and Figure 2, in particular Cam #1 and Cam #2), to obtain a three-dimensional spatial structure of the physical environment around the robotic arm (See at least “3D voxel maps” of Page 2, Section 1), wherein the intelligent obstacle-avoidance neural network model is a pre-trained (See at least caption of Figure 2, “The entire process does not involve any additional training”) intelligent model for performing dynamic three-dimensional reconstruction according to voxel data (See at least Page 4 and 5, Section 3.2, “Concretely, we aim to obtain a voxel value map Vt i = VoxPoser(ot,ℓi) by prompting an LLM and executing the code via a Python interpreter, where ot is the RGB-D observation at time t and ℓi is the current instruction”);
processing the three-dimensional spatial structure by adopting a robotic arm kinematics model, to obtain a motion trajectory of the robotic arm (See at least Page 6, Section 4, “Dynamics Model. We use the known robot dynamics model in all tasks, where it is used in motion planning for the end-effector to follow the waypoints” and/or Page 21, Section A.5, “The controller takes as input a desired end-effector 6-DoF pose, calculates a se quence of interpolated waypoints using inverse kinematics, and finally follows the waypoints using a PDcontroller”);
wherein controlling the robotic arm to sort the to-be-sorted part according to the control signal, the category and the grasp pose of the to-be-sorted part comprises:
controlling, according to the control signal, the category and the grasp pose of the to-be-sorted part, the robotic arm to sort the to-be-sorted part following the motion trajectory of the robotic arm (See at least Pages 3 and 4, Section 3.1, “The central problem investigated in this work is to generate a motion trajectory τri for robot r and each manipulation phase described by instruction ℓi. We represent τri as a sequence of dense end-effector waypoints to be executed by an Operational Space Controller [117], where each waypoint consists of a desired 6-DoF end-effector pose, end-effector velocity, and gripper action” and Figure 3 illustrating control for several tasks).
Regarding Claim 3, Huang teaches:
The method according to claim 1, wherein performing, according to the part sorting description information, the analysis processing by adopting the task instruction understanding model, to obtain the control signal for controlling the robotic arm to sort the to-be-sorted parts comprises:
performing understanding processing on the part sorting description information through a natural language understanding portion of the task instruction understanding model to obtain a feature and a sorting requirement of the to-be-sorted parts, and generating the control signal based on the feature and the sorting requirement of the to-be-sorted parts through a signal generation portion of the task instruction understanding model (See previous recitations and/or Figure 2 showing the flow of data and processing thereof. It is clear that natural language is provided, which is processed by an LLM which has “natural language understanding” portion(s) and is related to the part(s) in a scene and as shown with respect to Claim 1 has the features claimed. Furthermore, it is clear that signals are generated for control. In summary, this claim does not appear to significantly narrow from Claim 1 despite the length of the claim).
Regarding Claim 6, Huang teaches:
The method according to claim 2, wherein in the sorting process, processing, by adopting the intelligent obstacle-avoidance neural network model, the voxel data of different angles of the physical environment around the robotic arm which is collected in real time by the industrial camera, to obtain the three-dimensional spatial structure of the physical environment around the robotic arm comprises:
controlling multiple industrial cameras pre-arranged around the robotic arm to scan an activity space of the robotic arm at different angles and acquire the voxel data of different angles of the physical environment around the robotic arm (See at least Page 20, Section A.4, “We mount two RGB-D cameras (Azure Kinect) at two opposite ends of the table: bottom right and top left from the top down view. At the start of each rollout, both cameras start recording and return the real-time RGB-D observations at 20 Hz” and Figure 2, in particular Cam #1 and Cam #2);
inputting the voxel data of different angles of the physical environment around the robotic arm into the intelligent obstacle-avoidance neural network model to perform three-dimensional space feature extraction and perform three-dimensional space reconstruction according to an extracted feature, to obtain the three-dimensional space structure (See at least Page 2, Section 1, “By generating Python code to invoke perception APIs, LLMs can obtain spatial-geometric information of relevant objects or parts” and Page 6, Section 4, “VLMs and Perception. Given an object/part query from LLMs, we first invoke open-vocab detector OWL-ViT [15] to obtain a bounding box, then feed it into Segment Anything [118] to obtain a mask, and finally track the mask using video tracker XMEM [119]. The tracked mask is used with RGB-D observation to reconstruct the object/part point cloud”).
Regarding Claims 8 – 10, 13, 15 – 17, and 20, the claims are directed to effectively the same subject matter as Claims 1 – 3 and 6 with respect to the application of prior art. The claims are therefore rejected under the same logic as Claims 1 – 3 and 6 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 4 – 5, 11 – 12, and 18 – 19 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. further in view of Jie et al. (Jie, Shibo, and Zhi-Hong Deng. "Convolutional bypasses are better vision transformer adapters." arXiv preprint arXiv:2207.07039 (2022)).
Regarding Claim 4, Huang teaches:
The method according to claim 1, wherein performing, according to the control signal, the analysis processing on the image of the to-be-sorted part by adopting the intelligent perception model for parts, to obtain the category and the grasp pose of the to-be-sorted part, wherein the image is collected by the industrial camera, comprises:
controlling, according to the control signal, the industrial camera to acquire the image of the to-be-sorted part, wherein the image comprises an RGB image and a depth image (See at least Page 20, Section A.4, “We mount two RGB-D cameras (Azure Kinect) at two opposite ends of the table: bottom right and top left from the top down view. At the start of each rollout, both cameras start recording and return the real-time RGB-D observations at 20 Hz” and Figure 2, in particular Cam #1 and Cam #2);
inputting the RGB image and the depth image into the intelligent perception model for parts for performing feature analysis processing, to obtain the category and the grasp pose of the to-be-sorted part (See at least Page 4 and 5, Section 3.2, “Concretely, we aim to obtain a voxel value map Vt i = VoxPoser(ot,ℓi) by prompting an LLM and executing the code via a Python interpreter, where ot is the RGB-D observation at time t and ℓi is the current instruction”);
…
Huang does not explicitly teach the exact nature of the Vision Langue Model (VLM) utilized instead providing several references disclosing particular models, but Jie explicitly teaches a VLM of the claimed structure below:
…
wherein the intelligent perception model for parts comprises a Transformer layer (See at least “Transformer Layer” in Figure 3), a convolutional neural network layer (See at least “1x1 Conv” and “3x3 Conv” in Figure 3 and Page 3, Section 3.2, “As illustrated in Figure 3 , a Convpass module consists of three convolutional layers”) and a fully connected layer (See at least Page 2, Section 2.1, “An MLP block consists of two fully-connected (FC) lay-ers” and MLP in Figure 3).
It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the claimed invention to utilize the Vision Transformer Adapter of Jie, or at least features thereof, in the VLM of Huang with a reasonable expectation of success. The components claimed are well-known and routine, Huang already indicates that they are focused on leveraging existing understanding, and Jie indicates that “Convpass benefits from the hard-coded inductive bias of convolutional layers and thus is more suitable for visual tasks” (Abstract) when comparing to other Vision Transformers (ViT) which are utilized in VLMs.
Regarding Claim 5, the combination of Huang and Jie teaches:
The method according to claim 4,
Huang, in the context of the combination with Jie, further teaches:
wherein inputting the RGB image and the depth image into the intelligent perception model for parts for performing the feature analysis processing, to obtain the category and the grasp pose of the to-be-sorted part comprises:
performing feature extraction on the RGB image and the depth image through the Transformer layer of the intelligent perception model for parts to obtain a global visual feature of the to-be-sorted part;
performing feature extraction on the RGB image and the depth image through the convolutional neural network layer of the intelligent perception model for parts to obtain a local visual feature of the to-be-sorted part;
determining, according to the global visual feature and the local visual feature, the category and the grasp pose of the to-be-sorted part through the fully connected layer of the intelligent perception model for parts (See at least Page 2, Section 1, “By generating Python code to invoke perception APIs, LLMs can obtain spatial-geometric information of relevant objects or parts” and Page 6, Section 4, “VLMs and Perception. Given an object/part query from LLMs, we first invoke open-vocab detector OWL-ViT [15] to obtain a bounding box, then feed it into Segment Anything [118] to obtain a mask, and finally track the mask using video tracker XMEM [119]. The tracked mask is used with RGB-D observation to reconstruct the object/part point cloud”.
The combination of Huang and Jie inherently has all information in some manner passing through the VLM pass through the structures claimed in Claim 4, and the particular nature of “performing” or “determining” is not claimed with any particularity.
Furthermore, the natures of the “features” are not claimed with any particularity. A point cloud alone may be considered as inherently having both of these features, or the point cloud and mask may be considered these items, etc.).
Regarding Claims 11 – 12 and 18 – 19, the claims are directed to effectively the same subject matter as Claims 4 – 5 with respect to the application of prior art. The claims are therefore rejected under the same logic as Claims 4 – 5 above.
Allowable Subject Matter
Claims 7 and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The combination of limitations has not been found alone or in a reasonable combination of references found in Examiner’s search of the prior art. In particular, the limitation of (or similar) of Claim 7 which reads:
the fully connected layer is configured to reconstruct a grid model with a same size as an inputted voxel grid
has not been found in combination with the other limitations of the claims such that a reasonable rejection may be provided at this time. The closest prior art of record is that of Huang recited above, others recited in the conclusion section below which are considered to teach many or possibly all of the present independent claims, as well as Fazlazi et al. (US 20230281961 A1) which appears to teach the remaining limitations of these claims in combination with Huang or other references. Furthermore, Huang appears to at least teach reducing voxel maps (See Page 5, Section 3.2 of Huang, “Additionally, because V is often sparse, we densify the voxel maps via smoothing operations, as they encourage smoother trajectories optimized by motion planners”).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ding et al. (Ding, Yan, et al. "Task and motion planning with large language models for object rearrangement. In 2023 IEEE." RSJ International Conference on Intelligent Robots and Systems (IROS)) which discloses sorting items based on natural language instructions using LLMs.
Driess et al. (Driess, Danny, et al. "Palm-e: An embodied multimodal language model." arXiv preprint arXiv:2303.03378 (2023)) which discloses an embodied multimodal language model that is highly flexible to different user specified tasks and includes at least one specific example of sorting items per a user instruction to do so.
Florence et al. (US 20250144795 A1) which is a generally equivalent disclosure as Driess et al. as a patent publication.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW C GAMMON whose telephone number is (571)272-4919. The examiner can normally be reached M - F 10:00 - 6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ADAM MOTT can be reached on (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW C GAMMON/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
