DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The instant application claims priority to and benefit of Great Britian Application No. 2208343.0, filed on 06/07/2022. Thus, the effective filing date of Claims 1-20 is 06/07/2022.
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 11/18/2024 was reviewed and the listed references were noted.
Drawings
The 6 page drawings have been considered and placed on record in the file.
Status of Claims
Claims 1-20 are currently pending.
Claim Objections
Claim 17 is objected to because of the following informalities: in line 2 of the claim, the term “the an” should be replaced by “the”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art.
Ascertaining the differences between the prior art and the claims at issue.
Resolving the level of ordinary skill in the pertinent art.
Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kapelyukh (“My House, My Rules: Learning Tidying Preferences with Graph Neural Networks” published on November 4, 2021) in view of Batra (“Rearrangement: A Challenge for Embodied AI” published on November 3, 2020).
Consider Claim 1, Kapelyukh teaches “A method of determining an arrangement for objects,” (Kapelyukh; Abstract; “Given any set of objects, this vector can then be used to generate an arrangement which is tailored to that user’s spatial preferences”) “the method comprising: obtaining first data representing a first arrangement of first objects in a scene,” (Kapelyukh; Figure 2; Section 3.1; “Given a set of objects identified by s, and a user preference vector u, this predicts a position for each object which reflects those spatial preferences.”) “the first data comprising data representing the first objects and data representing the relative pose between the first objects;” (Kapelyukh; Figure 2 (See image below)) “inputting the obtained first data into a trained machine learning model to determine a first cost value for the first arrangement, the trained machine learning model having been trained to provide a cost function which, based on an input of data representing an arrangement of objects,” (Kapelyukh; Section 3.1; “Our objective is to learn a user encoder function Eφ, represented by a neural network with learned parameters φ. Its output is a user’s preference vector u. Its input is a representation of a scene arrangement made by that user.”) (Kapelyukh; Figure 4; “The input now consists of a set of objects for which we wish to predict positions, according to a user preference vector u. We set the node feature vectors to be xi pi u. This time, the output is a position vector for each node.”)
PNG
media_image1.png
326
929
media_image1.png
Greyscale
PNG
media_image2.png
465
935
media_image2.png
Greyscale
Kapelyukh does not explicitly disclose “…outputs a cost value indicative of an extent to which the arrangement differs from an optimum arrangement, wherein the first cost value is indicative of an extent to which the first arrangement differs from an optimum arrangement of the first objects according to the cost function;”. However, in an analogous field of endeavor, Batra teaches “…outputs a cost value indicative of an extent to which the arrangement differs from an optimum arrangement, wherein the first cost value is indicative of an extent to which the first arrangement differs from an optimum arrangement of the first objects according to the cost function;” (Batra; Section 4.1; “When an agent has finished working on a task, we must therefore determine our completion metric by comparing the final states of all objects with goal state s∗. …The placement accuracy of each object would be evaluated via a norm N(si,s∗ i) on the difference between each object’s final and target pose.”). Accordingly, before the effective filing date of the instant application, it would have been obvious to one of ordinary skill in the art to combine Kapelyukh with the teachings of Batra to further determine a quantitative number to measure the distance between objects and their desired position in an arrangement. One of ordinary skill in the art would be motivated to combine Kapelyukh and Batra to “obtain a mean accuracy measure” (Batra, Section 4.1). Accordingly, the combination of Kapelyukh and Batra discloses the invention of Claim 1.
Consider Claim 2, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein the method comprises: generating control instructions configured to cause a robot to move at least one of the first objects towards a pose that the at least one first object has in the second arrangement.” (Kapelyukh; Section 1; “For example, a domestic robot could be required to set a dinner table, tidy a messy desk, and find a home for a newly-bought object. Given goal states stipulating where each object should go, many approaches exist for planning a sequence of actions to manipulate objects into the desired arrangement”).
Consider Claim 3, the combination of Kapelyukh and Batra teaches “The method according to claim 2, wherein the method comprises: providing the control instructions to the robot to cause the robot to move at least one of the first objects towards a pose that the at least one first object has in the second arrangement.” (Kapelyukh; Section 2; “To tidy a scene, the robot computes a target tidy layout using a cost function which encourages altering the object-object distances in the untidy scene to match those in the “closest” tidy positive example.”)
Consider Claim 4, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein determining the second arrangement comprises
determining a second arrangement of the first objects that has a second cost value indicating that the second arrangement differs from the optimum arrangement to a lesser extent that the first arrangement.” (Kapelyukh; Figure 4 (See image above); “The input now consists of a set of objects for which we wish to predict positions, according to a user preference vector u.”).
Consider Claim 5, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein determining the second arrangement comprises
determining a gradient of the cost function at the first cost value.” (Kapelyukh; Equation 2 (See formula below)).
PNG
media_image3.png
82
841
media_image3.png
Greyscale
Consider Claim 6, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein determining the second arrangement comprises determining a minimum of the cost function.” (Kapelyukh; Section 4.2; “NeatNet was trained by passing training users through the Graph VAE and updating parameters with a neural network optimiser according to the loss function in Equation 3.1.” (emphasis added)).
Consider Claim 7, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein determining the second arrangement comprises
fixing the pose of one or more of the first objects of the first arrangement.” (Batra; Section 3.2; “The level of abstraction for the objects undergoing manipulation is another dimension of complexity. In the simplest case, manipulated objects are rigid bodies with no additional state that the agent can manipulate. Depending on the dexterity of the manipulator, the object may be rotated while it is held. The agent may have a stowing capacity in which case the object can be stowed away from the manipulator. A common abstraction is a virtual “backpack” with infinite or limited capacity stated in number of items, volume, or weight. More complex scenarios may involve object articulation states (e.g. books that may open and close).”). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 7 and are incorporated herein by reference. Thus, the method recited in claim 7 is met by Kapelyukh and Batra.
Consider Claim 8, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein determining the second arrangement based on the first cost value comprises: combining the first cost value with one or more further cost values for the first arrangement, thereby to generate a first combined cost value, the one or more further cost values each being determined from a respective further cost function and being indicative of a respective cost of the first arrangement as compared to a respective further optimum according to the respective further cost function;” (Batra; Section 4.2; “While multiple metrics can always be combined into single values via weighted addition or other formulae, this must be based on a choice of the relative importance of the different factors…”) “and determining the second arrangement based on the first combined cost value.” (Kapelyukh; Figure 4 (See image above); “The input now consists of a set of objects for which we wish to predict positions, according to a user preference vector u.”). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 8 and are incorporated herein by reference. Thus, the method recited in claim 8 is met by Kapelyukh and Batra.
Consider Claim 9, the combination of Kapelyukh and Batra teaches “The method according to claim 8, wherein the one or more further cost values comprise one or more of: an occupancy cost value indicative of an extent to which one or more of the first objects in the first arrangement occupies a space that is not to be occupied;
and a time cost value indicative of an estimate of a time it would take a robot to interact with one or more of the first objects in the first arrangement.” (Batra; Section 4.2; “Useful secondary metrics include: Simulation time taken for an agent to report completion and stop action: this is in units of the time defined within the simulation, or the number of simulation ‘ticks’ required, with respect to which the agent program can take actions at a constant defined rate.”). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 9 and are incorporated herein by reference. Thus, the method recited in claim 9 is met by Kapelyukh and Batra.
Consider Claim 10, the combination of Kapelyukh and Batra teaches “The method according to claim 8, wherein determining the second arrangement comprises:
determining a second arrangement that has a second combined cost value indicative of the second arrangement differing from a combination of the optimum arrangement and the respective further one or more optimums to a lesser extent that the first arrangement; and/or determining a gradient of a combined cost function at the first combined cost value, “the combined cost function being a combination of the cost function and the one or more further cost functions; and/or determining a minimum of a combined cost function,” (Batra; Section 4.2 ; “Later more sophisticated computational measures should also encompass the degree to which an agent’s computation and storage can be parallelized, distributed or layered in terms of latency.”) “the combined cost function being a combination of the cost function and the one or more further cost functions.” (Batra; Section 4.2; “While multiple metrics can always be combined into single values via weighted addition or other formulae, this must be based on a choice of the relative importance of the different factors…”). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 10 and are incorporated herein by reference. Thus, the method recited in claim 10 is met by Kapelyukh and Batra.
Consider Claim 11, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein the obtained first data comprises first graph data representing a graph representing the first arrangement of objects in the scene, the graph comprising nodes and edges connecting nodes,” (Kapelyukh; Figure 3 (See image below)) “wherein each node represents a respective object” (Kapelyukh; Figure 3 (See image below)) “and each edge represents a relative pose between two objects represented by two nodes that the edge connects,” (Kapelyukh; Section 3.2; “The scene graph is fully connected, to avoid making assumptions about which object-object relations are relevant or not.”) wherein the trained machine learning model is a trained graph neural network having been trained to provide a cost function which, based on an input of graph data representing a graph representing an arrangement of objects,” (Kapelyukh; Section 3.1; “Our objective is to learn a user encoder function Eφ, represented by a neural network with learned parameters φ. Its output is a user’s preference vector u. Its input is a representation of a scene arrangement made by that user.”) “outputs a cost value indicative of an extent to which the arrangement differs from an optimum arrangement.” (Batra; Section 4.1; “When an agent has finished working on a task, we must therefore determine our completion metric by comparing the final states of all objects with goal state s∗. …The placement accuracy of each object would be evaluated via a norm N(si,s∗ i) on the difference between each object’s final and target pose.”). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 11 and are incorporated herein by reference. Thus, the method recited in claim 11 is met by Kapelyukh and Batra.
PNG
media_image4.png
518
934
media_image4.png
Greyscale
Consider Claim 12, the combination of Kapelyukh and Batra teaches “The method according to claim 11, wherein the first graph data comprises: for each node of the graph a semantic vector representative of the respective object;” (Kapelyukh; Figure 3 (See image above); Section 3.2; “To do this, we use Graph Neural Network (GNN) layers….Each node represents an object in the scene. Node i has feature vector xi formed by concatenating the semantic embedding for that object si with its position encoding pi”) “and/or for each edge of the graph a relative pose vector representative of the relative pose between two objects represented by two nodes that the edge connects.” (Kapelyukh; Figure 3 (See image above)).
Consider Claim 13, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein the method comprises: obtaining image data representing an image of the objects of the scene in the first arrangement; and generating the first data based on the obtained image data.” (Kapelyukh; Figure 1 (See image below)).
PNG
media_image5.png
293
915
media_image5.png
Greyscale
Consider Claim 14, the combination of Kapelyukh and Batra teaches “The method according to claim 13, wherein the first graph data comprises: for each node of the graph a semantic vector representative of the respective object;” (Kapelyukh; Figure 3 (See image above); Semantic embedding) “and/or for each edge of the graph a relative pose vector representative of the relative pose between two objects represented by two nodes that the edge connects;” (Kapelyukh; Figure 3 (See image above)) “and wherein generating the first graph data comprises: generating the semantic vector for each of the one or more objects;” (Kapelyukh; Figure 3 (See image above); Semantic embedding) “and/or generating a pose vector for each of the one or more objects, the pose vector representing a pose of each of the objects,” (Kapelyukh; Figure 3 (See image above); Position encoding) “and determining the relative pose vector for each edge based on the pose vector for the two objects of the two respective nodes that the edge connects.” (Kapelyukh; Figure 3 (See image above)).
Consider Claim 15, the combination of Kapelyukh and Batra teaches “The method according to claim 1, wherein the method comprises training a machine learning model to provide the trained machine learning model, and wherein the training comprises: obtaining a training data set, the training data set comprising a plurality of sets of data, each set of data representing an arrangement of objects in a scene and comprising data representing the objects and data representing the relative pose between the objects,” (Kapelyukh; Section 4.2; “Rearrangement data from 75 users was gathered by distributing the experiment link on social media. Each user submits a tidy arrangement for a number of scenes. We set aside a group of 8 test users, male and female, and majority non-roboticists. The training dataset was formed from the 67 remaining users. NeatNet was trained by passing training users through the Graph VAE and updating parameters with a neural network optimiser according to the loss function in Equation 3.1. For each test user, we passed their example scenes through NeatNet to extract their spatial preferences, and predict a personalised arrangement for the test scene (varies by experiment).”) “wherein each set of data representing an arrangement of objects is associated with a cost value label indicative of the extent to which the arrangement differs from an optimum arrangement of the objects;” (Kapelyukh; Section 2; “Another approach is to ask each user to provide several tidy arrangements for a scene, to serve as positive examples [12]. To tidy a scene, the robot computes a target tidy layout using a cost function which encourages altering the object-object distances in the untidy scene to match those in the “closest” tidy positive example.” (emphasis added)) “and training the machine learning model, based on the training data set, to provide a cost function which, based on an input of data representing an arrangement of objects,” (Kapelyukh; Section 3.1; “Our objective is to learn a user encoder function Eφ, represented by a neural network with learned parameters φ. Its output is a user’s preference vector u. Its input is a representation of a scene arrangement made by that user.”) “outputs a cost value indicative of an extent to which the arrangement differs from an optimum arrangement.” (Batra; Section 4.1; “When an agent has finished working on a task, we must therefore determine our completion metric by comparing the final state s of all objects with goal state s∗….The placement accuracy of each object would be evaluated via a norm N(si,s∗ i) on the difference between each object’s final and target pose.”). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 15 and are incorporated herein by reference. Thus, the method recited in claim 15 is met by Kapelyukh and Batra.
Claim 16 recites a method with steps corresponding to the steps recited in Claim 15. Therefore, the recited steps of this claim are mapped to the proposed combination in the same manner as the corresponding steps in its corresponding method claim. Additionally, the rationale and motivation to combine the Kapelyukh and Batra references, presented in rejection of Claim 15, apply to this claim.
Consider Claim 17, the combination of Kapelyukh and Batra teaches “The method according to claim 16, wherein the training data set comprises a plurality of sets of graph data, each set of graph data representing a graph representing the an arrangement of objects in a scene,” (Kapelyukh; Figure 5 (See image below); “Learning one generalised user preference vector across multiple example scenes per user.”) “the graph comprising nodes and edges connecting nodes,” (Kapelyukh; Figure 3 (See image above)) “wherein each node represents a respective object and each edge represents a relative pose between two objects represented by two nodes that the edge connects,” (Kapelyukh; Figure 3 (See image above); Section 3.2; “The scene graph is fully connected, to avoid making assumptions about which object-object relations are relevant or not.”) “and wherein the machine learning model is a graph neural network.” (Kapelyukh; Section 3.2; “To do this, we use Graph Neural Network (GNN) layers.”).
PNG
media_image6.png
551
928
media_image6.png
Greyscale
Consider Claim 18, the combination of Kapelyukh and Batra teaches “An apparatus comprising: a processor; and a memory storing a computer program comprising a set of instructions which, when executed by the processor, cause the processor to perform the method according to claim 1.” (Batra; Pg. 20; Section A.2.; “A PC as the computing device: CPU Intel Xeon E 2246G, Memory 32GB DDR4, GPU Nvidia Geforce RTX2080 with 8GB memory.” (emphasis added)). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 18 and are incorporated herein by reference. Thus, the method recited in claim 18 is met by Kapelyukh and Batra.
Consider Claim 19, the combination of Kapelyukh and Batra teaches “The apparatus according to claim 18, wherein the apparatus is a robot configured to move one or more of the objects of the scene.” (Batra; Figure 7 (See image below); Manipulation). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 19 and are incorporated herein by reference. Thus, the method recited in claim 19 is met by Kapelyukh and Batra.
PNG
media_image7.png
368
998
media_image7.png
Greyscale
Consider Claim 20, the combination of Kapelyukh and Batra teaches “A non-transitory computer readable medium storing instructions which, when executed by a computer, cause the computer to perform the method according to claim 1.” (Batra; Pg. 20; Section A.2.; “A PC as the computing device: CPU Intel Xeon E 2246G, Memory 32GB DDR4, GPU Nvidia Geforce RTX2080 with 8GB memory.” (emphasis added)). The proposed combination as well as the motivation for combining the Kapelyukh and Batra references presented in the rejection of claim 1, apply to claim 20 and are incorporated herein by reference. Thus, the method recited in claim 20 is met by Kapelyukh and Batra.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Annie Pham whose telephone number is (571)272-1673. The examiner can be normally be reached Mon-Fri 9:00a – 5:00p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For
additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANNIE H PHAM/Examiner, Art Unit 2662
/Siamak Harandi/Primary Examiner, Art Unit 2662
Prosecution Insights