DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 01/17/2025, 06/26/2025, and 03/10/2026 have been considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claims 1, 3-8, 10, and 11 recite limitations that use words like “means” (or “step”) or similar terms with functional language and do invoke 35 U.S.C. 112(f):
Claim 1; recites the limitation, “obtaining, by an electronic device,…,” [Lines 3, 5, 8, 11].
Claim 1; recites the limitation, “updating, by the electronic device,…,” [Line 13].
Claim 3; recites the limitation, “analyzing, by the electronic device,…,” [Lines 3, 5].
Claim 4; recites the limitation, “the recognition model fails to recognize…,” [Line 5].
Claim 5; recites the limitation, “the recognition model will fail to recognize…,” [Line 2].
Claim 6; recites the limitation, “generating, by the electronic device,…,” [Lines 2, 7].
Claim 6; recites the limitation, “augmenting, by the electronic device,…,” [Line 4].
Claim 6; recites the limitation, “obtaining, by the electronic device,…,” [Line 5].
Claim 7; recites the limitation, “obtaining, by the electronic device,…,” [Line 2].
Claim 7; recites the limitation, “determining, by the electronic device,…,” [Lines 6, 9].
Claim 8; recites the limitation, “determining, by the electronic device,…,” [Line 5].
Claim 8; recites the limitation, “placing, by the electronic device,…,” [Line 8].
Claim 10; recites the limitation, “performing object recognition… using the recognition model;” [Lines 3-4].
Claim 11; recites the limitation, “performing, by the electronic device,…,” [Line 3].
Claim 11; recites the limitation, “requesting, by the electronic device,…,” [Line 5].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 1, 3-8, 10, and 11:
“electronic device” (Fig. 1, #100 called server, Paragraph [0188] – “the server 100 may perform fine- tuning on the recognition model included in an electronic device (the server 100) by using the training data.” Paragraph [0046] – “The server 100 may be an electronic device of various types having computational functions.” Thus, “electronic device” has sufficient structure wherein it is a server or computer.
“recognition model” (Fig. 3, #241 called recognition model, Paragraph [0031] – “As used herein, a "recognition model" may refer to a neural network model for recognizing objects included in an image or video.” Paragraph [0066] – “While FIG. 3 illustrates the processor 240 as including a recognition model 241, the recognition model 241 may refer to a neural network model implemented by the processor 240 executing a program stored in the memory 230.” Thus, “recognition model” has sufficient structure associated with it wherein it is a neural network model.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed
invention is not identically disclosed as set forth in section 102 of this title, if the
differences between the claimed invention and the prior art are such that the claimed
invention as a whole would have been obvious before the effective filing date of the
claimed invention to a person having ordinary skill in the art to which the claimed
invention pertains. Patentability shall not be negated by the manner in which the
invention was made.
Claims 1-3, 4, 6-8, 12-15, 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over AMER (US 20240160212 A1), hereinafter referenced as AMER in view of PRONOVOST (US 20240212360 A1), hereinafter referenced as PRONOVOST.
Regarding claim 1, AMER teaches a method of updating a recognition model of a robotic mobile device (Fig. 6, Paragraph [0106] – AMER discloses FIG. 6 illustrates an example of an environment navigation model updating method 600, performed in accordance with one or more embodiments. Paragraph [0080] – AMER further discloses the environment navigation model may be intermittently or continuously refined to improve navigation by the robot.), the method comprising:
obtaining, by an electronic device (Fig. 6, Paragraph [0106] – AMER discloses method 600 may be performed at one or more computing devices described herein, such as one or more devices shown in FIG. 2 or FIG. 3.),
from the robotic mobile device, spatial scan data regarding a target space (Fig. 6, Paragraph [0108] – AMER discloses sensor data is received at 606 via one or more sensors at the robot. According to various embodiments, the sensor data may include visual data collected from one or more cameras, depth scan information, accelerometer information, or any other data collected from the sensors at the robot.);
obtaining, by the electronic device (Fig. 6, Paragraph [0106] – AMER discloses method 600 may be performed at one or more computing devices described herein, such as one or more devices shown in FIG. 2 or FIG. 3.),
based on the spatial scan data (Figs. 6-7, Paragraph [0109] – AMER discloses at 608, the environment is navigated based on the sensor data and the environment navigation model. See also paragraph [0108].),
spatial information comprising information about a structure of the target space (Fig. 6-7, Paragraph [0117] – AMER discloses environment semantics are determined at 704. For example, environment semantics may identify features as characteristics of the environment. Such environmental features are typically fixed. For example, environment semantics may include ceilings, shelves, doors, windows, walls, floors, light fixtures, and/or other types of fixtures.)
and an item in the target space (Fig. 6-7, Paragraph [0118] – AMER discloses object semantics are determined at 704. According to various embodiments, object semantics may include characteristics of objects within the environment. For instance, object semantics may distinguish between a box and a fire extinguisher.);
and updating, by the electronic device (Fig. 17, Paragraph [0214] – AMER discloses the method 1700 may be performed on any suitable computing device or system.),
Although AMER further teaches the recognition model of the robotic mobile device using the training data (Fig. 17, Paragraph [0225] – AMER discloses the environment-specific object model is trained at 1716 based on the simulated sensor data, and then stored on a storage device. According to various embodiments, training the environment-specific object model may involve providing the simulated sensor data to a suitable machine learning model, such as a convolutional neural network or other deep learning model. See also Paragraph [0226].).
AMER fails to explicitly teach obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; obtaining, by the electronic device, training data by using the spatial information and the virtual object data;
However, PRONOVOST explicitly teaches obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object (Fig. 1, Paragraph [0031] – PRONOVOST discloses prediction component 104 can, for example, determine output data 108 representing a state of the vehicle 102, a state of various objects proximate the vehicle including an object 110 and an object 112, and/or scene data 114 usable for simulation. In various examples, the prediction component 104 can transmit the output data 108 to a planning component 116 for use during planning operations. Paragraph [0047] – PRONOVOST further discloses object state can indicate a position, orientation, velocity, acceleration, yaw; etc. of an object. Fig. 7, Paragraph [0083] – PRONOVOST further discloses the perception component 722 may include functionality to perform object detection, segmentation, and/or classification.)
by inputting the spatial information to a generative model (Fig. 2, Paragraph [0041] – PRONOVOST discloses example 200 includes a computing device (e.g., the vehicle computing device(s) 704 and/or the computing device(s) 734) that includes a generative model 202. In some examples, the generative model 202 can include at least the functionality of the prediction component 104 in FIG. 1. See also Paragraph [0036].);
obtaining, by the electronic device, training data by using the spatial information and the virtual object data (Fig. 7, Paragraph [0112] – PRONOVOST discloses the training component 748 may be executed by the processor(s) 736 to train a machine learning model based on training data. Paragraph [0114] – PRONOVOST further discloses the training component 748 can include training data that has been generated by a simulator.);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER of having a method of updating a recognition model of a robotic mobile device, the method comprising: obtaining, by an electronic device, from the robotic mobile device, spatial scan data regarding a target space; obtaining, by the electronic device, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space; and updating, by the electronic device, the recognition model of the robotic mobile device using the training data, with the teachings of PRONOVOST having obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; obtaining, by the electronic device, training data by using the spatial information and the virtual object data.
Wherein having AMER’s method of updating a recognition model of a robotic mobile device wherein having obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; obtaining, by the electronic device, training data by using the spatial information and the virtual object data.
The motivation behind the modification would have been to obtain an enhanced method of updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy and reduced computational resources, since both AMER and PRONOVOST relate to object recognition techniques, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and PRONOVOST describes techniques for applying and/or training one or more models to predict a representation of an object in an environment; data associated with navigation of the autonomous vehicle at a previous, current, or future time can be used to generate scene data that improves accuracy of determinations by the planning component over time given a fixed amount of available computational resources. Please see AMER (US 20240160212 A1), Paragraph [0033], and PRONOVOST (US 20240212360 A1), Paragraph [0014, 0017].
Regarding claim 2, AMER in view of PRONOVOST teach the method of claim 1,
AMER further teaches wherein the spatial scan data comprises an image of the target space captured by using a camera and map data obtained by scanning the target space using a light detection and ranging (LiDAR) sensor (Fig. 3, Paragraph [0041] – AMER discloses mechanisms described herein provide for a new approach for robot enrollment in a new environment, mapping, and remapping. According to various embodiments, the mapping may be initialized by any capture devices that use cameras, including RGB and RGB-D cameras on consumer phones and tablets. LiDAR sensor streams may be included as an additional, optional, source of information. Paragraph [0090] – AMER discloses types of sensors may include, but are not limited to: monocular visible light cameras, stereo visible light cameras, time-of-flight sensors, structured light cameras, depth sensors, LIDAR scanners, GPS interfaces, and accelerometers. For instance, may mobile computing devices are equipped with front and back cameras of various types, as well as other sensors such as accelerometers.).
Regarding claim 3, AMER in view of PRONOVOST teach the method of claim 2,
AMER further teaches wherein the obtaining the spatial information comprises: analyzing, by the electronic device, the structure of the target space based on the map data (Fig. 7, Paragraph [0117] – AMER discloses environment semantics are determined at 704. According to various embodiments, environment semantics may identify features as characteristics of the environment. Such environmental features are typically fixed. For example, environment semantics may include ceilings, shelves, doors, windows, walls, floors, light fixtures, and/or other types of fixtures. See also Fig. 6, Paragraph [0109-0110].);
and analyzing, by the electronic device, a class and a position of the item in the target space based on the map data and the image of the target space (Fig. 7, Paragraph [0118] – AMER discloses object semantics are determined at 704. According to various embodiments, object semantics may include characteristics of objects within the environment. For instance, object semantics may distinguish between a box and a fire extinguisher. Paragraph [0119] – AMER further discloses environment and/or object semantics may be determined based on any of a variety of types of information. For example, shape, color, and/or other visual information may be used to perform object recognition. As another example, objects may be observed at different points in time to determine whether or not they have moved.).
Regarding claim 4, AMER in view of PRONOVOST teach the method of claim 1,
AMER fails to explicitly teach wherein the generative model comprises a neural network model trained using a type of a failure event that occurred in the target space, a location where the failure event occurred, and the spatial information, and wherein the failure event comprises an event in which the recognition model fails to recognize an object.
However, PRONOVOST explicitly teaches wherein the generative model comprises a neural network model (Fig. 1, Paragraph [0036] – PRONOVOST discloses prediction component 104 may, in various examples, represent a generative machine learned model that is configured to receive (e.g., from a decoder of a variable autoencoder) occupancy information such as a point, contour, or bounding box associated with an object as the input data 106, and generate one or more scenes for use in a) a simulation between a vehicle and one or more objects proximate the vehicle, or b) a planning operation associated with a planning component.)
trained using a type of a failure event that occurred in the target space (Fig. 7, Paragraph [0094] – PRONOVOST discloses the machine learned algorithms may be trained to determine, based on sensor data and/or previous predictions by the model, that an object is likely to behave in a particular way relative to the vehicle 702 at a particular time during a set of estimated states (e.g., time period). Paragraph [0114] – PRONOVOST discloses for example, simulated training data can represent examples where a vehicle collides with an object in an environment or nearly collides with an object in an environment, to provide additional training examples.),
a location where the failure event occurred (Fig. 6, Paragraph [0074] – PRONOVOST discloses the training data 604 can include token data (e.g., a first token represents one of: a yield action, a drive straight action, a left turn action, a right turn action, a brake action, an acceleration action, a steering action, or a lane change action, and a second token represents a position, a heading, or an acceleration of the object), object state data associated with one or more objects (e.g., a previous trajectory, a previous action, a previous position, a previous acceleration, or other state or behavior of the object.) or vehicle state data associated with an autonomous vehicle.),
and the spatial information (Fig. 6, Paragraph [0074] – PRONOVOST discloses the training data 604 can include token data (e.g., a first token represents one of: a yield action, a drive straight action, a left turn action, a right turn action, a brake action, an acceleration action, a steering action, or a lane change action, and a second token represents a position, a heading, or an acceleration of the object), object state data associated with one or more objects (e.g., a previous trajectory, a previous action, a previous position, a previous acceleration, or other state or behavior of the object.) or vehicle state data associated with an autonomous vehicle. See also Paragraph [0040].),
and wherein the failure event comprises an event in which the recognition model fails to recognize an object (Fig. 6, Paragraph [0114] – PRONOVOST discloses for example, simulated training data can represent examples where a vehicle collides with an object in an environment or nearly collides with an object in an environment, to provide additional training examples.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having a method of updating a recognition model of a robotic mobile device, the method comprising: obtaining, by an electronic device, from the robotic mobile device, spatial scan data regarding a target space; obtaining, by the electronic device, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space; obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; with the teachings of PRONOVOST having wherein the generative model comprises a neural network model trained using a type of a failure event that occurred in the target space, a location where the failure event occurred, and the spatial information, and wherein the failure event comprises an event in which the recognition model fails to recognize an object.
Wherein having AMER’s method of updating a recognition model of a robotic mobile device wherein the generative model comprises a neural network model trained using a type of a failure event that occurred in the target space, a location where the failure event occurred, and the spatial information, and wherein the failure event comprises an event in which the recognition model fails to recognize an object.
The motivation behind the modification would have been to obtain an enhanced method of updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy and reduced computational resources, since both AMER and PRONOVOST relate to object recognition techniques, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and PRONOVOST describes techniques for applying and/or training one or more models to predict a representation of an object in an environment; data associated with navigation of the autonomous vehicle at a previous, current, or future time can be used to generate scene data that improves accuracy of determinations by the planning component over time given a fixed amount of available computational resources. Please see AMER (US 20240160212 A1), Paragraph [0033], and PRONOVOST (US 20240212360 A1), Paragraph [0014, 0017].
Regarding claim 6, AMER in view of PRONOVOST teach the method of claim 1,
AMER further teaches wherein the obtaining the training data comprises: generating, by the electronic device, a virtual space based on the spatial information (Fig. 17, Paragraph [0215] – AMER discloses a request to create an environment-specific object model for an object type is received at 1702. As discussed with respect to FIG. 15 and FIG. 16, the request may identify inputs such as sensor data of one or more physical instances of the object type, a three-dimensional model of the environment generated based on sensor input, and/or a three-dimensional model (e.g., a CAD model) of the object type.);
augmenting, by the electronic device, the virtual object in the virtual space (Fig. 16, Paragraph [0213] – AMER discloses at 1616, sensor data from the physical object, a 3D representation of the object, and/or environmental context information is combined via a scene renderer to generate an environment-specific representation of the object. The environment-specific representation of the object may be used to procedurally generate an image dataset for the object, which may then be used to train a model to recognize the object in various contexts and positions. See also Fig. 17, Paragraph [0222].);
obtaining, by the electronic device, a synthetic image of the virtual object captured in the virtual space by performing a simulation of the virtual space (Fig. 16, Paragraph [0213] – AMER discloses environment-specific representation of the object may be used to procedurally generate an image dataset for the object, which may then be used to train a model to recognize the object in various contexts and positions.);
and generating, by the electronic device, the training data using the synthetic image (Fig. 16, Paragraph [0213] – AMER discloses environment-specific representation of the object may be used to procedurally generate an image dataset for the object, which may then be used to train a model to recognize the object in various contexts and positions.).
Regarding claim 7, AMER in view of PRONOVOST teach the method of claim 6,
AMER further teaches wherein the obtaining the spatial information comprises obtaining, by the electronic device, illuminance characteristic information about the target space based on the spatial scan data (Fig. 17, Paragraph [0216] – AMER discloses one or more object parameters are determined at 1704. According to various embodiments, the object parameters may include intrinsic parameters, extrinsic parameters, placement parameters, and/or any other types of parameters. As yet another example, object extrinsic parameters may include aspects of the environment itself, such as light and shading under various conditions. Paragraph [0217] – AMER further discloses object parameters may be determined automatically, for instance by analyzing images or models of the object and/or the environment.),
and wherein the generating the virtual space comprises: determining, by the electronic device, a structure of the virtual space (Fig. 17, Paragraph [0030] – AMER discloses simulated sensor data of the object instances may then be generated based on simulated paths of a robot through the environment, as well as simulated sensors at the robot. Paragraph [0021] – AMER further discloses the system may procedurally change the object parameters to generate simulated images of the object within the environment. In particular embodiments, the system may place the objects in areas of the 3D environment where it is most likely to occur, and/or place the object randomly within the 3D environment.),
the class of the item (Fig. 17, Paragraph [0030] – AMER discloses simulated sensor data of the object instances may then be generated based on simulated paths of a robot through the environment, as well as simulated sensors at the robot. This simulated sensor data may be used to train the environment-specific object recognition model for recognizing instances of the object type [wherein object type is class of the item] within the physical environment.),
and the position of the item in the virtual space based on the spatial information (Fig. 17, Paragraph [0216] – AMER discloses one or more object parameters are determined at 1704. According to various embodiments, the object parameters may include intrinsic parameters, extrinsic parameters, placement parameters, and/or any other types of parameters. For example, intrinsic parameters may include aspects of the object itself that may vary depending on the particular instance of the object, such as size, shape, texture, shading, stacking, and blur. As another example, object placement parameters may include aspects of how the object is located within the environment, such as how the object is likely to be grouped, rotated, stacked vertically, and/or stacked horizontally.);
and determining, by the electronic device, illuminance for each of a plurality of regions in the virtual space based on the illuminance characteristic information (Fig. 17, Paragraph [0216] – AMER discloses one or more object parameters are determined at 1704. According to various embodiments, the object parameters may include intrinsic parameters, extrinsic parameters, placement parameters, and/or any other types of parameters. As yet another example, object extrinsic parameters may include aspects of the environment itself, such as light and shading under various conditions. Paragraph [0217] – AMER further discloses object parameters may be determined automatically, for instance by analyzing images or models of the object and/or the environment.).
Regarding claim 8, AMER in view of PRONOVOST teach the method of claim 7,
AMER further teaches wherein the virtual object data further comprises context information regarding nearby items related to the virtual object (Fig. 17, Paragraph [0216] – AMER discloses object placement parameters may include aspects of how the object is located within the environment, such as how the object is likely to be grouped, rotated, stacked vertically, and/or stacked horizontally.),
and wherein the augmenting the virtual object comprises: determining, by the electronic device, a position in the virtual space where the virtual object is to be placed based on the item in the virtual space and the context information (Fig. 17, Paragraph [0220] – PRONOVOST discloses the object parameter values may include any or all of the extrinsic, intrinsic, and placement parameters. The object parameter values may be selected in various combinations, to ensure that images of the object in the environment are simulated using various possible appearances of the object. For example, the parameter values may be varied to ensure that images of the object are generated for various object positions, stacking configurations, lighting conditions, and/or angles. In addition, a combination may be used for more than one simulated path, to ensure that the object is viewed from different perspectives.);
and placing, by the electronic device, the virtual object at the determined position in the virtual space (Fig. 17, Paragraph [0216] – AMER discloses object placement parameters may include aspects of how the object is located within the environment, such as how the object is likely to be grouped, rotated, stacked vertically, and/or stacked horizontally.).
Regarding claim 12, AMER teaches an electronic device for updating a recognition model of a robotic mobile device (Fig. 14, Paragraph [0203] – AMER discloses FIG. 14 illustrates one example of a computing device, configured in accordance with one or more embodiments.), the electronic device comprising:
memory storing a program or at least one instruction (Fig. 14, Paragraph [0203] – AMER discloses a system 1400 suitable for implementing embodiments described herein includes a processor 1401, a memory module 1403.);
and at least one processor operatively coupled to the memory (Fig. 14, Paragraph [0203] – AMER discloses a system 1400 suitable for implementing embodiments described herein includes a processor 1401, a memory module 1403.),
wherein the program or the at least one instruction, when executed by the at least one processor (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.),
causes the electronic device (Fig. 14, Paragraph [0203] – AMER discloses FIG. 14 illustrates one example of a computing device.) to:
obtain spatial scan data regarding a target space from the robotic mobile device (Fig. 6, Paragraph [0108] – AMER discloses sensor data is received at 606 via one or more sensors at the robot. According to various embodiments, the sensor data may include visual data collected from one or more cameras, depth scan information, accelerometer information, or any other data collected from the sensors at the robot.),
obtain, based on the spatial scan data (Figs. 6-7, Paragraph [0109] – AMER discloses at 608, the environment is navigated based on the sensor data and the environment navigation model. See also paragraph [0108].), spatial information comprising information about a structure of the target space (Fig. 6-7, Paragraph [0117] – AMER discloses environment semantics are determined at 704. For example, environment semantics may identify features as characteristics of the environment. Such environmental features are typically fixed. For example, environment semantics may include ceilings, shelves, doors, windows, walls, floors, light fixtures, and/or other types of fixtures.)
and an item in the target space (Fig. 6-7, Paragraph [0118] – AMER discloses object semantics are determined at 704. According to various embodiments, object semantics may include characteristics of objects within the environment. For instance, object semantics may distinguish between a box and a fire extinguisher.),
Although AMER further teaches and update the recognition model of the robotic mobile device using the training data (Fig. 17, Paragraph [0225] – AMER discloses the environment-specific object model is trained at 1716 based on the simulated sensor data, and then stored on a storage device. According to various embodiments, training the environment-specific object model may involve providing the simulated sensor data to a suitable machine learning model, such as a convolutional neural network or other deep learning model. See also Paragraph [0226].).
AMER fails to explicitly teach obtain virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model, obtain training data by using the spatial information and the virtual object data,
However, PRONOVOST explicitly teaches obtain virtual object data comprising information about a class of a virtual object and a position of the virtual object (Fig. 1, Paragraph [0031] – PRONOVOST discloses prediction component 104 can, for example, determine output data 108 representing a state of the vehicle 102, a state of various objects proximate the vehicle including an object 110 and an object 112, and/or scene data 114 usable for simulation. In various examples, the prediction component 104 can transmit the output data 108 to a planning component 116 for use during planning operations. Paragraph [0047] – PRONOVOST further discloses object state can indicate a position, orientation, velocity, acceleration, yaw; etc. of an object. Fig. 7, Paragraph [0083] – PRONOVOST further discloses the perception component 722 may include functionality to perform object detection, segmentation, and/or classification.)
by inputting the spatial information to a generative model (Fig. 2, Paragraph [0041] – PRONOVOST discloses example 200 includes a computing device (e.g., the vehicle computing device(s) 704 and/or the computing device(s) 734) that includes a generative model 202. In some examples, the generative model 202 can include at least the functionality of the prediction component 104 in FIG. 1. See also Paragraph [0036].),
obtain training data by using the spatial information and the virtual object data (Fig. 7, Paragraph [0112] – PRONOVOST discloses the training component 748 may be executed by the processor(s) 736 to train a machine learning model based on training data. Paragraph [0114] – PRONOVOST further discloses the training component 748 can include training data that has been generated by a simulator.),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER of having an electronic device for updating a recognition model of a robotic mobile device, the electronic device comprising: memory storing a program or at least one instruction; and at least one processor operatively coupled to the memory, wherein the program or the at least one instruction, when executed by the at least one processor, causes the electronic device to: obtain spatial scan data regarding a target space from the robotic mobile device, obtain, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space, and update the recognition model of the robotic mobile device using the training data, with the teachings of PRONOVOST having obtain virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model, obtain training data by using the spatial information and the virtual object data.
Wherein having AMER’s electronic device for updating a recognition model of a robotic mobile device wherein having obtain virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model, obtain training data by using the spatial information and the virtual object data.
The motivation behind the modification would have been to obtain an enhanced device for updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy and reduced computational resources, since both AMER and PRONOVOST relate to object recognition techniques, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and PRONOVOST describes techniques for applying and/or training one or more models to predict a representation of an object in an environment; data associated with navigation of the autonomous vehicle at a previous, current, or future time can be used to generate scene data that improves accuracy of determinations by the planning component over time given a fixed amount of available computational resources. Please see AMER (US 20240160212 A1), Paragraph [0033], and PRONOVOST (US 20240212360 A1), Paragraph [0014, 0017].
Regarding claim 13, AMER in view of PRONOVOST teach the electronic device of claim 12,
AMER further teaches wherein the spatial scan data comprises an image of the target space captured by using a camera and map data obtained by scanning the target space using a light detection and ranging (LiDAR) sensor (Fig. 3, Paragraph [0041] – AMER discloses mechanisms described herein provide for a new approach for robot enrollment in a new environment, mapping, and remapping. According to various embodiments, the mapping may be initialized by any capture devices that use cameras, including RGB and RGB-D cameras on consumer phones and tablets. LiDAR sensor streams may be included as an additional, optional, source of information. Paragraph [0090] – AMER discloses types of sensors may include, but are not limited to: monocular visible light cameras, stereo visible light cameras, time-of-flight sensors, structured light cameras, depth sensors, LIDAR scanners, GPS interfaces, and accelerometers. For instance, may mobile computing devices are equipped with front and back cameras of various types, as well as other sensors such as accelerometers.).
Regarding claim 14, AMER in view of PRONOVOST teach the electronic device of claim 13,
AMER further teaches wherein the program or the at least one instruction, when executed by the at least one processor (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.),
cause the electronic device (Fig. 14, Paragraph [0203] – AMER discloses FIG. 14 illustrates one example of a computing device.)
to obtain the spatial information by: analyzing a structure of the target space based on the map data (Fig. 7, Paragraph [0117] – AMER discloses environment semantics are determined at 704. According to various embodiments, environment semantics may identify features as characteristics of the environment. Such environmental features are typically fixed. For example, environment semantics may include ceilings, shelves, doors, windows, walls, floors, light fixtures, and/or other types of fixtures. See also Fig. 6, Paragraph [0109-0110].),
and analyzing, based on the map data and the image of the target space, a class and a position of the item in the target space (Fig. 7, Paragraph [0118] – AMER discloses object semantics are determined at 704. According to various embodiments, object semantics may include characteristics of objects within the environment. For instance, object semantics may distinguish between a box and a fire extinguisher. Paragraph [0119] – AMER further discloses environment and/or object semantics may be determined based on any of a variety of types of information. For example, shape, color, and/or other visual information may be used to perform object recognition. As another example, objects may be observed at different points in time to determine whether or not they have moved.).
Regarding claim 15, AMER in view of PRONOVOST teach the electronic device of claim 12,
AMER fails to explicitly teach wherein the generative model comprises a neural network model trained using a type of a failure event that occurred in the target space, a location where the failure event occurred, and the spatial information, and wherein the failure event comprises an event in which the recognition model fails to recognize an object.
However, PRONOVOST explicitly teaches wherein the generative model comprises a neural network model (Fig. 1, Paragraph [0036] – PRONOVOST discloses prediction component 104 may, in various examples, represent a generative machine learned model that is configured to receive (e.g., from a decoder of a variable autoencoder) occupancy information such as a point, contour, or bounding box associated with an object as the input data 106, and generate one or more scenes for use in a) a simulation between a vehicle and one or more objects proximate the vehicle, or b) a planning operation associated with a planning component.)
trained using a type of a failure event that occurred in the target space (Fig. 7, Paragraph [0094] – PRONOVOST discloses the machine learned algorithms may be trained to determine, based on sensor data and/or previous predictions by the model, that an object is likely to behave in a particular way relative to the vehicle 702 at a particular time during a set of estimated states (e.g., time period). Paragraph [0114] – PRONOVOST discloses for example, simulated training data can represent examples where a vehicle collides with an object in an environment or nearly collides with an object in an environment, to provide additional training examples.),
a location where the failure event occurred (Fig. 6, Paragraph [0074] – PRONOVOST discloses the training data 604 can include token data (e.g., a first token represents one of: a yield action, a drive straight action, a left turn action, a right turn action, a brake action, an acceleration action, a steering action, or a lane change action, and a second token represents a position, a heading, or an acceleration of the object), object state data associated with one or more objects (e.g., a previous trajectory, a previous action, a previous position, a previous acceleration, or other state or behavior of the object.) or vehicle state data associated with an autonomous vehicle.),
and the spatial information (Fig. 6, Paragraph [0074] – PRONOVOST discloses the training data 604 can include token data (e.g., a first token represents one of: a yield action, a drive straight action, a left turn action, a right turn action, a brake action, an acceleration action, a steering action, or a lane change action, and a second token represents a position, a heading, or an acceleration of the object), object state data associated with one or more objects (e.g., a previous trajectory, a previous action, a previous position, a previous acceleration, or other state or behavior of the object.) or vehicle state data associated with an autonomous vehicle. See also Paragraph [0040].),
and wherein the failure event comprises an event in which the recognition model fails to recognize an object (Fig. 6, Paragraph [0114] – PRONOVOST discloses for example, simulated training data can represent examples where a vehicle collides with an object in an environment or nearly collides with an object in an environment, to provide additional training examples.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having an electronic device for updating a recognition model of a robotic mobile device, the electronic device comprising: memory storing a program or at least one instruction; and at least one processor operatively coupled to the memory, wherein the program or the at least one instruction, when executed by the at least one processor, causes the electronic device to: obtain virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model, with the teachings of PRONOVOST having wherein the generative model comprises a neural network model trained using a type of a failure event that occurred in the target space, a location where the failure event occurred, and the spatial information, and wherein the failure event comprises an event in which the recognition model fails to recognize an object.
Wherein having AMER’s electronic device for updating a recognition model of a robotic mobile device wherein the generative model comprises a neural network model trained using a type of a failure event that occurred in the target space, a location where the failure event occurred, and the spatial information, and wherein the failure event comprises an event in which the recognition model fails to recognize an object.
The motivation behind the modification would have been to obtain an enhanced device for updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy and reduced computational resources, since both AMER and PRONOVOST relate to object recognition techniques, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and PRONOVOST describes techniques for applying and/or training one or more models to predict a representation of an object in an environment; data associated with navigation of the autonomous vehicle at a previous, current, or future time can be used to generate scene data that improves accuracy of determinations by the planning component over time given a fixed amount of available computational resources. Please see AMER (US 20240160212 A1), Paragraph [0033], and PRONOVOST (US 20240212360 A1), Paragraph [0014, 0017].
Regarding claim 17, AMER in view of PRONOVOST teach the electronic device of claim 12,
AMER further teaches wherein the program or the at least one instruction, when executed by the at least one processor (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.),
cause the electronic device (Fig. 14, Paragraph [0203] – AMER discloses FIG. 14 illustrates one example of a computing device.)
to obtain the training data by: generating a virtual space based on the spatial information (Fig. 17, Paragraph [0215] – AMER discloses a request to create an environment-specific object model for an object type is received at 1702. As discussed with respect to FIG. 15 and FIG. 16, the request may identify inputs such as sensor data of one or more physical instances of the object type, a three-dimensional model of the environment generated based on sensor input, and/or a three-dimensional model (e.g., a CAD model) of the object type.),
augmenting the virtual object in the virtual space (Fig. 16, Paragraph [0213] – AMER discloses at 1616, sensor data from the physical object, a 3D representation of the object, and/or environmental context information is combined via a scene renderer to generate an environment-specific representation of the object. The environment-specific representation of the object may be used to procedurally generate an image dataset for the object, which may then be used to train a model to recognize the object in various contexts and positions. See also Fig. 17, Paragraph [0222].),
obtaining a synthetic image of the virtual object captured in the virtual space by performing a simulation on the virtual space (Fig. 16, Paragraph [0213] – AMER discloses environment-specific representation of the object may be used to procedurally generate an image dataset for the object, which may then be used to train a model to recognize the object in various contexts and positions.),
and generating the training data by using the synthetic image (Fig. 16, Paragraph [0213] – AMER discloses environment-specific representation of the object may be used to procedurally generate an image dataset for the object, which may then be used to train a model to recognize the object in various contexts and positions.).
Regarding claim 18, AMER in view of PRONOVOST teach the electronic device of claim 17,
AMER further teaches wherein the program or the at least one instruction, when executed by the at least one processor (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.),
cause the electronic device (Fig. 14, Paragraph [0203] – AMER discloses FIG. 14 illustrates one example of a computing device.) to:
obtain illuminance characteristic information about the target space based on the spatial scan data (Fig. 17, Paragraph [0216] – AMER discloses one or more object parameters are determined at 1704. According to various embodiments, the object parameters may include intrinsic parameters, extrinsic parameters, placement parameters, and/or any other types of parameters. As yet another example, object extrinsic parameters may include aspects of the environment itself, such as light and shading under various conditions. Paragraph [0217] – AMER further discloses object parameters may be determined automatically, for instance by analyzing images or models of the object and/or the environment.),
and generate the virtual space by: determining, based on the spatial information, a structure of the virtual space (Fig. 17, Paragraph [0030] – AMER discloses simulated sensor data of the object instances may then be generated based on simulated paths of a robot through the environment, as well as simulated sensors at the robot. Paragraph [0021] – AMER further discloses the system may procedurally change the object parameters to generate simulated images of the object within the environment. In particular embodiments, the system may place the objects in areas of the 3D environment where it is most likely to occur, and/or place the object randomly within the 3D environment.)
and the class (Fig. 17, Paragraph [0030] – AMER discloses simulated sensor data of the object instances may then be generated based on simulated paths of a robot through the environment, as well as simulated sensors at the robot. This simulated sensor data may be used to train the environment-specific object recognition model for recognizing instances of the object type [wherein object type is class of the item] within the physical environment.)
and the position of the item in the virtual space (Fig. 17, Paragraph [0216] – AMER discloses one or more object parameters are determined at 1704. According to various embodiments, the object parameters may include intrinsic parameters, extrinsic parameters, placement parameters, and/or any other types of parameters. For example, intrinsic parameters may include aspects of the object itself that may vary depending on the particular instance of the object, such as size, shape, texture, shading, stacking, and blur. As another example, object placement parameters may include aspects of how the object is located within the environment, such as how the object is likely to be grouped, rotated, stacked vertically, and/or stacked horizontally.),
and determine illuminance for each of a plurality of regions in the virtual space based on the illuminance characteristic information (Fig. 17, Paragraph [0216] – AMER discloses one or more object parameters are determined at 1704. According to various embodiments, the object parameters may include intrinsic parameters, extrinsic parameters, placement parameters, and/or any other types of parameters. As yet another example, object extrinsic parameters may include aspects of the environment itself, such as light and shading under various conditions. Paragraph [0217] – AMER further discloses object parameters may be determined automatically, for instance by analyzing images or models of the object and/or the environment.).
Regarding claim 19, AMER in view of PRONOVOST teach the electronic device of claim 18,
AMER further teaches wherein the virtual object data further comprises context information regarding nearby items related to the virtual object (Fig. 17, Paragraph [0216] – AMER discloses object placement parameters may include aspects of how the object is located within the environment, such as how the object is likely to be grouped, rotated, stacked vertically, and/or stacked horizontally.),
and wherein the program or the at least one instruction, when executed by the at least one processor (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.),
cause the electronic device (Fig. 14, Paragraph [0203] – AMER discloses FIG. 14 illustrates one example of a computing device.) to:
augment the virtual object by: determining, based on the item in the virtual space and the context information, a position in the virtual space where the virtual object is to be placed (Fig. 17, Paragraph [0220] – PRONOVOST discloses the object parameter values may include any or all of the extrinsic, intrinsic, and placement parameters. The object parameter values may be selected in various combinations, to ensure that images of the object in the environment are simulated using various possible appearances of the object. For example, the parameter values may be varied to ensure that images of the object are generated for various object positions, stacking configurations, lighting conditions, and/or angles. In addition, a combination may be used for more than one simulated path, to ensure that the object is viewed from different perspectives.),
and placing the virtual object at the determined position in the virtual space (Fig. 17, Paragraph [0216] – AMER discloses object placement parameters may include aspects of how the object is located within the environment, such as how the object is likely to be grouped, rotated, stacked vertically, and/or stacked horizontally.).
Regarding claim 20, AMER teaches a system for updating a recognition model (Fig. 2, Paragraph [0081] – AMER discloses FIG. 2 illustrates an architecture diagram of a robot management system 200, configured in accordance with one or more embodiments.), the system comprising:
a robotic mobile device (Fig. 2, Paragraph [0083] – AMER discloses the devices shown in the system 200 may include robots, mobile enrollment devices, or both. See also Figs. 3 & 9.) comprising:
at least one robotic mobile device memory storing at least one robotic mobile device instruction (Fig. 9, Paragraph [0141] – AMER discloses robotic cart 800 includes a processor 902, a memory module 904, a communication interface 906, a storage device 908. See also Paragraph [0142].);
a robotic mobile device communication interface (Fig. 9, Paragraph [0143] – AMER discloses the robotic cart 800 may include one or more communication interfaces 906 configured to perform wired and/or wireless communication.);
a camera (Fig. 9, Paragraph [0144] – AMER discloses the sensor module 910 may include one or more of various types of sensors. Such sensors may include, but are not limited to: visual light cameras, infrared cameras, microphones, Lidar devices, Radar devices, chemical detection devices, near field communication devices, and accelerometers.);
a light detection and ranging (LiDAR) sensor (Fig. 9, Paragraph [0144] – AMER discloses the sensor module 910 may include one or more of various types of sensors. Such sensors may include, but are not limited to: visual light cameras, infrared cameras, microphones, Lidar devices, Radar devices, chemical detection devices, near field communication devices, and accelerometers.);
and at least one robotic mobile device processor configured to execute the at least one robotic mobile device instruction (Fig. 9, Paragraph [0142] – AMER discloses the robotic cart 800 may include one or more processors 902 configured to perform operations described herein.);
and a server (Fig. 14, Paragraph [0203] – AMER discloses a system 1400 suitable for implementing embodiments described herein. AMER further discloses system 1400 may operate as variety of devices such as robot, remote server, or any other device or service described herein.) comprising:
at least one server memory storing at least one server instruction (Fig. 14, Paragraph [0203] – AMER discloses system 1400 suitable for implementing embodiments described herein includes a processor 1401, a memory module 1403, a storage device 1405, an interface 1411, and a bus 1415.);
a server communication interface (Fig. 14, #1411 called interface, Paragraph [0203]);
and at least one server processor configured to execute the at least one server instruction (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.),
wherein the at least one robotic mobile device instruction, when executed by the at least one robotic mobile device processor (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.), causes the robotic mobile device to:
obtain spatial scan data comprising an image of a target space captured using the camera and map data obtained by scanning the target space using the LiDAR sensor (Fig. 6, Paragraph [0108] – AMER discloses sensor data is received at 606 via one or more sensors at the robot. According to various embodiments, the sensor data may include visual data collected from one or more cameras, depth scan information, accelerometer information, or any other data collected from the sensors at the robot. Paragraph [0090] – AMER discloses types of sensors may include, but are not limited to: monocular visible light cameras, stereo visible light cameras, time-of-flight sensors, structured light cameras, depth sensors, LIDAR scanners, GPS interfaces, and accelerometers. See also Paragraph [0111].),
and transmit, through the robotic mobile device communication interface (Fig. 9, #906 called communication interfaces, Paragraph [0143]), the spatial scan data to the server (Fig. 6, Paragraph [0111] – AMER discloses updating the environment navigation model may involve implementing a Visual simultaneous localization and mapping (SLAM) process in which the position and orientation of one or more sensors at the robot are determined with respect to the environment, while simultaneously mapping the environment around the sensor.),
and wherein the at least one server instruction, when executed by the at least one server processor (Fig. 14, Paragraph [0203] – AMER discloses processor 1401 may perform operations such as those described herein. Instructions for performing such operations may be embodied in the memory 1403, on one or more non-transitory computer readable media, or on some other storage device.), causes the server to:
obtain, through the server communication interface (Fig. 14, #1411 called interface, Paragraph [0203]), the spatial scan data from the robotic mobile device (Fig. 6, Paragraph [0108] – AMER discloses sensor data is received at 606 via one or more sensors at the robot. According to various embodiments, the sensor data may include visual data collected from one or more cameras, depth scan information, accelerometer information, or any other data collected from the sensors at the robot.),
obtain, based on the spatial scan data (Figs. 6-7, Paragraph [0109] – AMER discloses at 608, the environment is navigated based on the sensor data and the environment navigation model. See also paragraph [0108].),
spatial information comprising information about a structure of the target space (Fig. 6-7, Paragraph [0117] – AMER discloses environment semantics are determined at 704. For example, environment semantics may identify features as characteristics of the environment. Such environmental features are typically fixed. For example, environment semantics may include ceilings, shelves, doors, windows, walls, floors, light fixtures, and/or other types of fixtures.)
and an item in the target space (Fig. 6-7, Paragraph [0118] – AMER discloses object semantics are determined at 704. According to various embodiments, object semantics may include characteristics of objects within the environment. For instance, object semantics may distinguish between a box and a fire extinguisher.),
Although AMER further teaches and cause the robotic mobile device to update the recognition model using the training data (Fig. 17, Paragraph [0225] – AMER discloses the environment-specific object model is trained at 1716 based on the simulated sensor data, and then stored on a storage device. According to various embodiments, training the environment-specific object model may involve providing the simulated sensor data to a suitable machine learning model, such as a convolutional neural network or other deep learning model. See also Paragraph [0226].).
AMER fails to explicitly teach obtain virtual object data comprising information about a class and a position of a virtual object by inputting the spatial information to a generative model, obtain training data by using the spatial information and the virtual object data,
However, PRONOVOST explicitly teaches obtain virtual object data comprising information about a class and a position of a virtual object (Fig. 1, Paragraph [0031] – PRONOVOST discloses prediction component 104 can, for example, determine output data 108 representing a state of the vehicle 102, a state of various objects proximate the vehicle including an object 110 and an object 112, and/or scene data 114 usable for simulation. In various examples, the prediction component 104 can transmit the output data 108 to a planning component 116 for use during planning operations. Paragraph [0047] – PRONOVOST further discloses object state can indicate a position, orientation, velocity, acceleration, yaw; etc. of an object. Fig. 7, Paragraph [0083] – PRONOVOST further discloses the perception component 722 may include functionality to perform object detection, segmentation, and/or classification.)
by inputting the spatial information to a generative model (Fig. 2, Paragraph [0041] – PRONOVOST discloses example 200 includes a computing device (e.g., the vehicle computing device(s) 704 and/or the computing device(s) 734) that includes a generative model 202. In some examples, the generative model 202 can include at least the functionality of the prediction component 104 in FIG. 1. See also Paragraph [0036].),
obtain training data by using the spatial information and the virtual object data (Fig. 7, Paragraph [0112] – PRONOVOST discloses the training component 748 may be executed by the processor(s) 736 to train a machine learning model based on training data. Paragraph [0114] – PRONOVOST further discloses the training component 748 can include training data that has been generated by a simulator.),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER of having a system for updating a recognition model, the system comprising: a robotic mobile device comprising: at least one robotic mobile device memory storing at least one robotic mobile device instruction; a robotic mobile device communication interface; a camera; a light detection and ranging (LiDAR) sensor; and at least one robotic mobile device processor configured to execute the at least one robotic mobile device instruction; and a server comprising: at least one server memory storing at least one server instruction; a server communication interface; and at least one server processor configured to execute the at least one server instruction, wherein the at least one robotic mobile device instruction, when executed by the at least one robotic mobile device processor, causes the robotic mobile device to: obtain spatial scan data comprising an image of a target space captured using the camera and map data obtained by scanning the target space using the LiDAR sensor, and transmit, through the robotic mobile device communication interface, the spatial scan data to the server, and wherein the at least one server instruction, when executed by the at least one server processor, causes the server to: obtain, through the server communication interface, the spatial scan data from the robotic mobile device, obtain, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space, and cause the robotic mobile device to update the recognition model using the training data, with the teachings of PRONOVOST having obtain virtual object data comprising information about a class and a position of a virtual object by inputting the spatial information to a generative model, obtain training data by using the spatial information and the virtual object data.
Wherein having AMER’s system for updating a recognition model wherein having obtain virtual object data comprising information about a class and a position of a virtual object by inputting the spatial information to a generative model, obtain training data by using the spatial information and the virtual object data.
The motivation behind the modification would have been to obtain an enhanced system for updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy and reduced computational resources, since both AMER and PRONOVOST relate to object recognition techniques, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and PRONOVOST describes techniques for applying and/or training one or more models to predict a representation of an object in an environment; data associated with navigation of the autonomous vehicle at a previous, current, or future time can be used to generate scene data that improves accuracy of determinations by the planning component over time given a fixed amount of available computational resources. Please see AMER (US 20240160212 A1), Paragraph [0033], and PRONOVOST (US 20240212360 A1), Paragraph [0014, 0017].
Claims 5 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over AMER (US 20240160212 A1), hereinafter referenced as AMER in view of PRONOVOST (US 20240212360 A1), hereinafter referenced as PRONOVOST in further view of YAO (US 20220114825 A1), hereinafter referenced as YAO.
Regarding claim 5, AMER in view of PRONOVOST teach the method of claim 1,
AMER in view of PRONOVOST fail to explicitly teach wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
However, YAO explicitly teaches wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold (Fig. 1, Paragraph [0010] – YAO discloses hard examples, as used herein, may refer to example images that may have a higher probability of being a false positive or a false negative. In some examples, the hard examples may include both positive and negative hard examples. For example, hard positive example images may be images containing an object that may have a higher probability of not being detected as an object. Paragraph [0021] – YAO further discloses sample scoring and mining module 106 may then select hard example sample candidates from all candidate boxes according to their multi-task loss score. For example, sample candidates with multi-task loss scores above a threshold score may be selected to be used for training.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having a method of updating a recognition model of a robotic mobile device, the method comprising: obtaining, by an electronic device, from the robotic mobile device, spatial scan data regarding a target space; obtaining, by the electronic device, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space; obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; with the teachings of YAO having wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
Wherein having AMER’s method of updating a recognition model of a robotic mobile device wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
The motivation behind the modification would have been to obtain an enhanced method of updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy, since both AMER and YAO relate to arrangements for image or video recognition/understanding, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and YAO discloses an apparatus, method and system for training neural networks using multi-scale hard example mining; using hard examples of both positive examples and negative examples may improve detection accuracy, while also reducing the number of example images to be used to train a CNN. Please see AMER (US 20240160212 A1), Paragraph [0033], and YAO (US 20220114825 A1), Paragraph [0011, 0046].
Regarding claim 16, AMER in view of PRONOVOST teach the electronic device of claim 12,
AMER in view of PRONOVOST fail to explicitly teach wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
However, YAO explicitly teaches wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold (Fig. 1, Paragraph [0010] – YAO discloses hard examples, as used herein, may refer to example images that may have a higher probability of being a false positive or a false negative. In some examples, the hard examples may include both positive and negative hard examples. For example, hard positive example images may be images containing an object that may have a higher probability of not being detected as an object. Paragraph [0021] – YAO further discloses sample scoring and mining module 106 may then select hard example sample candidates from all candidate boxes according to their multi-task loss score. For example, sample candidates with multi-task loss scores above a threshold score may be selected to be used for training.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having an electronic device for updating a recognition model of a robotic mobile device, the electronic device comprising: memory storing a program or at least one instruction; and at least one processor operatively coupled to the memory, wherein the program or the at least one instruction, when executed by the at least one processor, causes the electronic device to: obtain virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model, with the teachings of YAO having wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
Wherein having AMER’s electronic device for updating a recognition model of a robotic mobile device wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
The motivation behind the modification would have been to obtain an enhanced device for updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy, since both AMER and YAO relate to arrangements for image or video recognition/understanding, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and YAO discloses an apparatus, method and system for training neural networks using multi-scale hard example mining; using hard examples of both positive examples and negative examples may improve detection accuracy, while also reducing the number of example images to be used to train a CNN. Please see AMER (US 20240160212 A1), Paragraph [0033], and YAO (US 20220114825 A1), Paragraph [0011, 0046].
Claims 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over AMER (US 20240160212 A1), hereinafter referenced as AMER in view of PRONOVOST (US 20240212360 A1), hereinafter referenced as PRONOVOST in further view of VENKATARAMAN (US 20220215266 A1), hereinafter referenced as VENKATARAMAN.
Regarding claim 9, AMER in view of PRONOVOST teach the method of claim 6,
AMER in view of PRONOVOST fail to explicitly teach wherein the generating the training data using the synthetic image comprises labeling the synthetic image with the class of the virtual object.
However, VENKATARAMAN explicitly teaches wherein the generating the training data using the synthetic image comprises labeling the synthetic image with the class of the virtual object (Fig. 1, Paragraph [0154] – VENKATARAMAN discloses when generating training data for supervised learning, the synthetic data generator 40 also automatically generates labels (e.g., desired outputs) for the synthesized images. For example, when generating training data for training a machine learning model to perform an image classification task, the generated label for a given image may include the classes of the objects depicted in the image.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having a method of updating a recognition model of a robotic mobile device, the method comprising: obtaining, by an electronic device, from the robotic mobile device, spatial scan data regarding a target space; obtaining, by the electronic device, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space; obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; with the teachings of VENKATARAMAN having wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
Wherein having AMER’s method of updating a recognition model of a robotic mobile device wherein the virtual object is an object for which a probability that the recognition model will fail to recognize in the target space is greater than or equal to a preset threshold.
The motivation behind the modification would have been to obtain an enhanced method of updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved efficiency, since both AMER and VENKATARAMAN relate to aspects of machine learning, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and VENKATARAMAN discloses machine learning techniques, in particular the synthesis or generation of data for training machine learning models; using hard example mining to sample the synthesized data sets can improve the efficiency of the training process by reducing the size of the training set to remove substantially redundant images that would not have much impact on the training process while keeping the “hard examples” that have more of an impact on the resulting trained model. Please see AMER (US 20240160212 A1), Paragraph [0033], and VENKATARAMAN (US 20220215266 A1), Paragraph [0153].
Regarding claim 10, AMER in view of PRONOVOST teach the method of claim 6,
AMER in view of PRONOVOST fail to explicitly teach wherein the generating the training data using the synthetic image comprises: performing object recognition on the synthetic image using the recognition model; determining whether the object recognition is successful; determining whether to use the synthetic image as the training data based on a result of the determining whether the object recognition is successful; and based on determining to use the synthetic image as the training data, generating the training data by labeling the synthetic image with the class of the virtual object.
However, VENKATARAMAN explicitly teaches wherein the generating the training data using the synthetic image (Fig. 1, Paragraph [0154] – VENKATARAMAN discloses when generating training data for supervised learning, the synthetic data generator 40 also automatically generates labels (e.g., desired outputs) for the synthesized images. For example, when generating training data for training a machine learning model to perform an image classification task, the generated label for a given image may include the classes of the objects depicted in the image.) comprises:
performing object recognition on the synthetic image using the recognition model; determining whether the object recognition is successful (Fig. 1, Paragraph [0153] – VENKATARAMAN discloses the images for the training data set are sampled from the synthesized data sets (1), (2), and (3) based on hard example mining. Using hard example mining to sample the synthesized data sets can improve the efficiency of the training process by reducing the size of the training set to remove substantially redundant images that would not have much impact on the training process while keeping the “hard examples” that have more of an impact on the resulting trained model.);
determining whether to use the synthetic image as the training data based on a result of the determining whether the object recognition is successful (Fig. 1, Paragraph [0153] – VENKATARAMAN discloses the images for the training data set are sampled from the synthesized data sets (1), (2), and (3) based on hard example mining. Using hard example mining to sample the synthesized data sets can improve the efficiency of the training process by reducing the size of the training set to remove substantially redundant images that would not have much impact on the training process while keeping the “hard examples” that have more of an impact on the resulting trained model.);
and based on determining to use the synthetic image as the training data, generating the training data by labeling the synthetic image with the class of the virtual object (Fig. 1, Paragraph [0154] – VENKATARAMAN discloses when generating training data for supervised learning, the synthetic data generator 40 also automatically generates labels (e.g., desired outputs) for the synthesized images. For example, when generating training data for training a machine learning model to perform an image classification task, the generated label for a given image may include the classes of the objects depicted in the image. These classification label may be generated by identifying each unique type of object that is visible in the virtual scene.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having a method of updating a recognition model of a robotic mobile device, the method comprising: obtaining, by an electronic device, from the robotic mobile device, spatial scan data regarding a target space; obtaining, by the electronic device, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space; obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; with the teachings of VENKATARAMAN having wherein the generating the training data using the synthetic image comprises: performing object recognition on the synthetic image using the recognition model; determining whether the object recognition is successful; determining whether to use the synthetic image as the training data based on a result of the determining whether the object recognition is successful; and based on determining to use the synthetic image as the training data, generating the training data by labeling the synthetic image with the class of the virtual object.
Wherein having AMER’s method of updating a recognition model of a robotic mobile device wherein the generating the training data using the synthetic image comprises: performing object recognition on the synthetic image using the recognition model; determining whether the object recognition is successful; determining whether to use the synthetic image as the training data based on a result of the determining whether the object recognition is successful; and based on determining to use the synthetic image as the training data, generating the training data by labeling the synthetic image with the class of the virtual object.
The motivation behind the modification would have been to obtain an enhanced method of updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved efficiency, since both AMER and VENKATARAMAN relate to aspects of machine learning, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and VENKATARAMAN discloses machine learning techniques, in particular the synthesis or generation of data for training machine learning models; using hard example mining to sample the synthesized data sets can improve the efficiency of the training process by reducing the size of the training set to remove substantially redundant images that would not have much impact on the training process while keeping the “hard examples” that have more of an impact on the resulting trained model. Please see AMER (US 20240160212 A1), Paragraph [0033], and VENKATARAMAN (US 20220215266 A1), Paragraph [0153].
Regarding claim 11, AMER in view of PRONOVOST teach the method of claim 1,
AMER fails to explicitly teach wherein the updating the recognition model of the robotic mobile device comprises: and requesting, by the electronic device, an update of the recognition model of the robotic mobile device by transmitting, to the robotic mobile device, parameter information of the recognition model of the electronic device on which the fine-tuning has been performed.
However, PRONOVOST explicitly teaches wherein the updating the recognition model of the robotic mobile device comprises: and requesting, by the electronic device, an update of the recognition model of the robotic mobile device by transmitting, to the robotic mobile device, parameter information of the recognition model of the electronic device on which the fine-tuning has been performed (Fig. 1 & 10, Paragraph [0032] – PRONOVOST discloses the prediction component 104 can transmit the output data 108 to a planning component 116 for use during planning operations. For example, the planning component 116 can determine a vehicle trajectory 118 for the scene 114. Paragraph [0133] – PRONOVOST further discloses the training component 602 can compare the first output data or the second output data to ground truth and train the encoder or the decoder based at least in part on the comparison. In various examples, data associated with a trained machine learned model can be transmitted to a vehicle computing device for use in controlling an autonomous vehicle in an environment.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having a method of updating a recognition model of a robotic mobile device, the method comprising: obtaining, by an electronic device, from the robotic mobile device, spatial scan data regarding a target space; obtaining, by the electronic device, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space; obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; with the teachings of PRONOVOST having wherein the updating the recognition model of the robotic mobile device comprises: and requesting, by the electronic device, an update of the recognition model of the robotic mobile device by transmitting, to the robotic mobile device, parameter information of the recognition model of the electronic device on which the fine-tuning has been performed.
Wherein having AMER’s method of updating a recognition model of a robotic mobile device wherein the updating the recognition model of the robotic mobile device comprises: and requesting, by the electronic device, an update of the recognition model of the robotic mobile device by transmitting, to the robotic mobile device, parameter information of the recognition model of the electronic device on which the fine-tuning has been performed.
The motivation behind the modification would have been to obtain an enhanced method of updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved accuracy and reduced computational resources, since both AMER and PRONOVOST relate to object recognition techniques, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and PRONOVOST describes techniques for applying and/or training one or more models to predict a representation of an object in an environment; data associated with navigation of the autonomous vehicle at a previous, current, or future time can be used to generate scene data that improves accuracy of determinations by the planning component over time given a fixed amount of available computational resources. Please see AMER (US 20240160212 A1), Paragraph [0033], and PRONOVOST (US 20240212360 A1), Paragraph [0014, 0017].
AMER in view of PRONOVOST fail to explicitly teach performing, by the electronic device, fine-tuning on a recognition model of the electronic device using the training data;
However, VENKATARAMAN explicitly teaches performing, by the electronic device, fine-tuning on a recognition model of the electronic device using the training data (Fig. 1, Paragraph [0063] – VENKATARAMAN discloses model training system 7 may apply an iterative process for updating the parameters of the model 30 to generate the trained model 32 in accordance with the supplied training data 5 (e.g., including the synthesized data 42). The updating of the parameters of the model 30 may include, for example, applying gradient descent (and, in the case of a neural network, backpropagation) in accordance with a loss function measuring a difference between the labels and the output of the model in response to the training data.);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of AMER in view of PRONOVOST of having a method of updating a recognition model of a robotic mobile device, the method comprising: obtaining, by an electronic device, from the robotic mobile device, spatial scan data regarding a target space; obtaining, by the electronic device, based on the spatial scan data, spatial information comprising information about a structure of the target space and an item in the target space; obtaining, by the electronic device, virtual object data comprising information about a class of a virtual object and a position of the virtual object by inputting the spatial information to a generative model; with the teachings of VENKATARAMAN having performing, by the electronic device, fine-tuning on a recognition model of the electronic device using the training data.
Wherein having AMER’s method of updating a recognition model of a robotic mobile device wherein having performing, by the electronic device, fine-tuning on a recognition model of the electronic device using the training data.
The motivation behind the modification would have been to obtain an enhanced method of updating a recognition model of a robotic device capable of object detection in different environmental contexts with improved efficiency, since both AMER and VENKATARAMAN relate to aspects of machine learning, wherein AMER relates generally to robotics, and more specifically to recognition of objects in a physical environment by a robot; techniques and mechanisms described herein may be applied to improve applications such as inventory tracking, localization, object counting, object management, and/or labor management, and VENKATARAMAN discloses machine learning techniques, in particular the synthesis or generation of data for training machine learning models; using hard example mining to sample the synthesized data sets can improve the efficiency of the training process by reducing the size of the training set to remove substantially redundant images that would not have much impact on the training process while keeping the “hard examples” that have more of an impact on the resulting trained model. Please see AMER (US 20240160212 A1), Paragraph [0033], and VENKATARAMAN (US 20220215266 A1), Paragraph [0153].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
ANWAR et al. (US 20240312123 A1) - In various examples, systems and methods are disclosed that relate to data augmentation for training/updating perception models in autonomous or semi-autonomous systems and applications. For example, a system may receive data associated with a set of frames that are captured using a plurality of cameras positioned in fixed relation relative to the machine; generate a panoramic view based at least on the set of frames; provide data associated with the panoramic view to a model to cause the model to generate a high dynamic range (HDR) panoramic view; determine lighting information associated with a light distribution map based at least on the HDR panoramic view; determine a virtual scene; and render an asset and a shadow on at least one of the frames, based at least on the virtual scene and the light distribution map, the shadow being a shadow corresponding to the asset.… Fig. 1, Abstract.
EBRAHIMI AFROUZI et al. (US 20240310851 A1) - A method for operating a robot, including: capturing images of a workspace; capturing data indicative of movement of the robot; capturing LIDAR data as the robot moves within the workspace; generating a map of the workspace based on the LIDAR data; actuating the robot to drive; discriminating between an object on a floor surface along a path of the robot and the floor surface based on the captured images; actuating the robot to drive until determining all areas of the workspace are discovered and included in the map; and executing a cleaning function..… Fig. 1, Abstract.
GOYAL et al. (US 20230234233 A1) - Apparatuses, systems, and techniques to place one or more objects in a location and orientation. In at least one embodiment, one or more circuits are to use one or more neural networks to cause one or more autonomous devices to place one or more objects in a location and orientation based, at least in part, on one or more images of the location and orientation..… Fig. 1, Abstract.
PARK et al. (US 20220139086 A1) - According to the present invention, disclosed are a device and a method of generating an object image, recognizing an object, and learning an environment of a mobile robot which perform a deep learning algorithm which allows a robot to create a map and load environment information acquired during the autonomous movement while the autonomous mobile robot is being charged and may be used for an application which finds out a location by finally recognizing objects such as furniture using a method of checking a location of the recognized objects to mark the location on the map...… Fig. 1, Abstract.
DUQUETTE et al. (US 12675903 B2) - A method of generating a synthetic image for producing a dataset of synthetic images for training an artificial intelligence model for object recognition; it includes generating a virtual space including a synthetic image background comprising one or more synthetic surfaces; spawning one or more virtual objects in the virtual space; simulating freefall and/or simulating environmental conditions of the one or more virtual objects resulting in a simulated impact between the one or more virtual objects and the one or more synthetic surfaces for determining the shape and position of the one or more virtual objects on the synthetic image background; and generating the synthetic image composed of the synthetic image background and the one or more virtual objects with the position of the one or more virtual objects on the synthetic image background determined from the simulated impact and/or simulated environmental conditions....… Fig. 1, Abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEZAWIT N SHIMELES whose telephone number is (571)272-7663. The examiner can normally be reached M-F 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEZAWIT NOLAWI SHIMELES/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights