DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ebstyne et al. (U.S. Pub. No. 20190347372) in view of Trigoni et al. (U.S. Pub. No. 20170241787), further in view of Kurz et al. “Camera Motion Style Transfer”, 11/2010, IEEE.
Regarding claim 1, Ebstyne discloses the method comprising (para 80, “FIG. 8 is a flow chart 800 illustrating an example work flow of a computer vision and speech design service suitable for implementing some of the various examples disclosed herein.”; also, para 92, “Other examples are directed to developing a computer vision or speech solution, the method comprising: importing, into a simulation, one or more hardware configurations for a sensor platform comprising one or more virtual sensors; generating an environment simulation for one or more virtual environments; generating a motion profile simulating motion of the one or more hardware configurations within the one or more virtual environments; generating synthetic experiment data for the one or more hardware configurations having the simulated motion within the one or more virtual environments; and iterating the generation of synthetic experiment data for one or more combinations of hardware configurations, virtual environment, motion and computer vision or speech algorithms.”): generating simulated ground truth data (para 76, “Evaluator 636 may be used for comparing the calculated computer vision and speech data with ground truth data.”; also, para 76, “Because environment orchestrator 406 and motion orchestrator 404 produce the output for simulator 624 to use, the information about where the synthetic DUT is within the synthetic environment is available for use in the evaluation process.”; also, para 76, “This known data, actual position of DUT and objects) [sic] is collectively known as the ground truth (GT) 638.”; also, para 80, “Generating synthetic experiment data comprises simulating, for one or more hardware configurations sensor data that can be supplied to a computer vision and speech algorithm.”) based on a modified trajectory (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”) of a second device (para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs.”; also, para 52, “That is, the computing device 302 may represent a real-world device that is designed using an end-to-end computer vision and speech design service, or may represent a synthetic version used as a test candidate for data generation.”; also, para 52, “In some examples, a mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, wearable device, head mounted display (HMD) and/or portable media player.”) of a second user (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 31, “Just as a real-world camera may be moved around a room by users, capturing video of a room to generate data for a synthesized version of that room, a synthetic camera placed virtually into a synthetic scene can generate an equivalent data set.”) in a second physical environment (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 20, “Setting up an experimentation workflow for a manual build-test-repeat type process typically requires dependence on manually collecting and labelling large amounts of data that must be collected by sending testers with prototype devices into real-world environments to collect sensor data streams.”), retraining a computer vision algorithm with the simulated ground truth data (para 78, “For these lesser-accurate computer vision and speech applications, disparity data of the simulated virtual hardware configuration compared with its GT data 702 may be fed back to the algorithm application 710 to improve the performance of such computer vision and speech applications.”). Ebstyne does not disclose determining a gait pattern of a first user of a first device in a first physical environment, the gait pattern determined by a machine-learning model trained by an unsupervised training algorithm using inertial measurement unit data from an inertial sensor of the first device and poses of the first device over time in the first physical environment; the modified trajectory adapting poses of the second device over time to the gait pattern of the first user.
However, in a similar field of endeavor, Trigoni discloses determining a gait pattern of a first user of a first device in a first physical environment (para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”; also, para 154, “Different step constant parameters were learned for different pedestrians.”; also, para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”) , the gait pattern determined by a machine-learning model trained by an unsupervised training algorithm (para 138, “To address this issue, to the present embodiment uses an unsupervised approach to learning R-PDR parameters.”; also, para 132, “This is referred to here as unsupervised lifelong learning approach as LL-Tracker, and describe in detail later below.”; also, para 139, “The solution to this optimization can be obtained with the expectation maximization (EM) approach.”) using inertial measurement unit data from an inertial sensor of the first device (para 99, “However, tracking device orientation remains a major issue with low-cost inertial measurement unit (IMU) sensors embedded in mobile devices that are not constrained to be held in a certain way, which IMU sensors include a gyro and a magnetometer for the purpose.”; also, para 82, “The accelerometer, magnetometer and gyroscope are the main sensors but others could be taken into account by the system to provide position, acceleration and orientation signals, such as a barometer, light sensors and so on.”) and poses of the first device over time in the first physical environment (para 85, “The output of the R-PDR layer 10, a trajectory for the user, in particular in this embodiment it comprises a distance value and heading value for each step taken by the user of the device.”; also, para 107, “As noted above, the gravity vector estimates are then fed into the Kalman filter as additional observations (along with magnetometer measurements) to estimate the orientation of the device.”; also, para 139, “S(x) is the matched trajectory and Z(x) is the raw trajectory fed from R-PDR to map matching.”); the gait pattern of the first user (para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 154, “Different step constant parameters were learned for different pedestrians.”; also, para 132, “The accuracy of this model is not satisfactory by itself because step length parameters vary across different users and environments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ebstyne's invention of a work flow that generates ground truth data and simulated sensor data for one or more hardware configurations, that simulates the trajectory and orientation of a synthetic device under test from motion modeled or recorded from actual real-world devices captured by a user walking around a real-world scene, and that feeds the disparity between the simulated data and the ground truth data back to improve the computer vision applications, with the features of Trigoni's invention of learning, by an unsupervised approach using expectation maximization, a per-individual step constant from the inertial measurement unit sensors embedded in a carried smart phone together with the trajectory and the estimated orientation of that device. The combination would have been obvious because Ebstyne already captures both of the data streams Trigoni learns from, stating that inertial measurement unit data and data from other six degree of freedom device sensors may be captured by a user walking around a real-world scene, yet Ebstyne stops at using that capture as a template for typical movement scenarios and never characterizes how the particular person who carried the device walks. Trigoni supplies exactly that missing characterization, and supplies it without labels, because it states that step length parameters vary across different users and that different step constant parameters were learned for different pedestrians. A person of ordinary skill working on Ebstyne's pipeline and wanting its simulated motion to reflect a particular user rather than a generic walk would have applied Trigoni's unsupervised learning to the inertial and trajectory streams Ebstyne already collects, with the predictable result of a per-user walking characterization derived from the same capture Ebstyne performs. The characterization so learned belongs to the particular first user whose device supplied the data, because Trigoni states that different step constant parameters were learned for different pedestrians and that step length parameters vary across different users, so it is that user's gait pattern to which the second device's poses are adapted.
Kurz discloses the modified trajectory adapting poses of the second device over time (sec 3.1 “The resulting camera motion is a regularly-sampled time series of camera positions and orientations. Such a representation has six degrees of freedom per frame, where position is stored as an Euclidean 3-vector x and orientation as a unit quaternion q.”; also, sec 1, “Therefore, in this paper we present an approach to transfer camera shake from real videos onto an artificially designed camera motion path in a virtual scene.”; also, sec 4, “First, the user selects a simple walking motion from a certain movie as style. Our system transfers this walking style to a different camera animation.”; also, sec 3.1, “From this data, we reconstruct the three-dimensional camera motion paths originally used in the shooting of the footage.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Trigoni, in which a per-user walking characterization is learned without labels from a carried device's inertial sensors and its trajectory and orientation while a motion orchestrator simulates the trajectory and orientation of a device under test, with the features of Kurz's invention of transferring a walking motion recovered from a real recorded camera path onto a different camera path whose every frame carries six degrees of freedom of position and orientation. The combination would have been obvious because the base combination produces the walking characterization and the device trajectory as two separate quantities and leaves open how one is imposed on the other, which is the precise operation Kurz performs. A person of ordinary skill would have used Kurz's transfer to impose the learned per-user walking characterization on the motion orchestrator's simulated trajectory, with the predictable result that the simulated ground truth data reflects how that particular user walks rather than a generic random walk.
Regarding claim 2, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 1, wherein Ebstyne further comprising: identifying a trajectory (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”) of the second device (para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs.”; also, para 52, “That is, the computing device 302 may represent a real-world device that is designed using an end-to-end computer vision and speech design service, or may represent a synthetic version used as a test candidate for data generation.”; also, para 52, “In some examples, a mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, wearable device, head mounted display (HMD) and/or portable media player.”) of the second user (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 31, “Just as a real-world camera may be moved around a room by users, capturing video of a room to generate data for a synthesized version of that room, a synthetic camera placed virtually into a synthetic scene can generate an equivalent data set.”) in the second physical environment (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 20, “Setting up an experimentation workflow for a manual build-test-repeat type process typically requires dependence on manually collecting and labelling large amounts of data that must be collected by sending testers with prototype devices into real-world environments to collect sensor data streams.”); applying the modified trajectory to a plurality of virtual environments (para 64, “Environment orchestrator 406 is used for simulating one or more virtual environments. In some embodiments, environment orchestrator 406 permits users to manipulate synthetic environments, such as light settings and the state of certain objects, such as doors.”; also, para 63, “Motion orchestrator 404 may be used for simulating motion of the one or more simulated hardware configurations within one or more virtual environments. Examples may include creating instances of multiple random walks through a virtual scene or room, having various durations, speeds, and motion pathways.”). Ebstyne does not disclose determining the modified trajectory of the second device based on the gait pattern of the first user.
However, in a similar field of endeavor, Kurz discloses determining the modified trajectory of the second device (sec 4, “First, the user selects a simple walking motion from a certain movie as style. Our system transfers this walking style to a different camera animation. “; also, sec 3.1, “The resulting camera motion is a regularly-sampled time series of camera positions and orientations. Such a representation has six degrees of freedom per frame, where position is stored as an Euclidean 3-vector x and orientation as a unit quaternion q. “; also, abstract, “Consequently, an arbitrary virtual base motion, defined in any conventional animation package, can be automatically modified according to a user-selected style.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Trigoni, in which a motion orchestrator simulates the trajectory and orientation of a device under test through one or more virtual environments and a per-user walking characterization is learned from a carried device's inertial and trajectory data, with the features of Kurz's invention of transferring a walking motion selected from a recorded camera path onto a different camera animation. The combination would have been obvious because Ebstyne's motion orchestrator already accepts a motion profile as its input and already drives the device under test through a plurality of virtual environments, so a person of ordinary skill needed only a way to express the learned walking characterization as that profile, and Kurz supplies a worked technique for imposing a walking motion recovered from one camera path onto another.
Trigoni discloses based on the gait pattern of the first user (para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 154, ‘Different step constant parameters were learned for different pedestrians.”; also, para 132, “The accuracy of this model is not satisfactory by itself because step length parameters vary across different users and environments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Kurz, in which a walking characterization is imposed on the motion orchestrator's simulated trajectory, with the features of Trigoni's invention of learning a step constant separately for each individual pedestrian from that pedestrian's own carried device. The combination would have been obvious because the characterization Kurz imposes has to be measured from something, and Trigoni already supplies it as a per-person quantity, stating that different step constant parameters were learned for different pedestrians and that step length parameters vary across different users and environments. A person of ordinary skill would therefore have conditioned the determination on the walking characterization learned from the first user's own device rather than on a generic walk, with the predictable result that the modified trajectory reflects that individual's walk.
Regarding claim 3, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 2, wherein Ebstyne further discloses the first device comprises a first visual tracking device (para 52, “In some examples, a mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, wearable device, head mounted display (HMD) and/or portable media player.”; also, para 58, “As illustrated, computing device 302 further includes a camera 330 (though other types of sensors may be used), which may represent a single camera, a stereo camera set, a set of differently-facing cameras, or another configuration.”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”), wherein the second device comprises a second visual tracking device (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs. Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”), wherein the trajectory is based on poses of the second visual tracking device over time (para 58, “The combination of 3D position and 3D rotation may be referred to as six degrees-of-freedom (6DoF), and a combination of 3D accelerometer and 3D gyroscope data may permit 6DoF measurements.”; also, para 81, “In step 806, some number, M, of different device positions and orientations are generated.”; also, para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”).
Regarding claim 4, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 1, wherein Ebstyne further discloses the modified trajectory is applied to a plurality of virtual environments (para 63, “Motion orchestrator 404 may be used for simulating motion of the one or more simulated hardware configurations within one or more virtual environments. Examples may include creating instances of multiple random walks through a virtual scene or room, having various durations, speeds, and motion pathways.”; also, para 64, “Environment orchestrator 406 is used for simulating one or more virtual environments.”).
Regarding claim 5, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 1, wherein Ebstyne further discloses the retrained computer vision algorithm is configured for a motion pattern of the first user (para 78, “In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 67, “For instance, a computer vision and speech application that produced computer vision and speech algorithm output data comprising computer vision and speech results that exceeded a certain variance threshold (e.g., more than X percentage away from the GT of a simulated hardware configuration) away from the underlying GT may be used by an artificial intelligence (AI) application running in the cloud environment 200 to optimize the deficient computer vision and speech application by running more testing against other synthetic scenes, motions, and hardware configurations until the computer vision and speech application performs within the variance threshold.”; also, para 83, “Step 814a iterates on various different candidate computer vision and speech algorithms; step 8144 iterates on various different motion profiles; step 814c iterates on various different environments and EOCs; and step 814d iterates on various different candidate hardware configurations.”; also, para 63, “In some embodiments, motion orchestrator module 404 permits users of computer vision and speech design service 400 to model motion that is relevant for testing computer vision and speech sensor platforms and algorithms by expressing targeted motion profiles.”).
Regarding claim 6, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 5, wherein Ebstyne further comprising: providing the retrained computer vision algorithm to the first device (para 78, “In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 56, “For example, the applications may represent downloaded client-side applications that correspond to server-side services executing in a cloud.”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”), wherein the first device is re-configured for the motion pattern of the first user based on the retrained computer vision algorithm (para 84, “Experimental data is evaluated in step 816, for example by comparing computer vision and speech algorithm output data with GT data, and a recommendation step 818 may provide a recommended best performing configuration or data from which a designer may select an adequately performing configuration.”; also, para 84, “In this manner hardware configurations may be selected from a set of candidate hardware configurations, computer vision and speech algorithms may be selected from a set of candidate algorithms, or computer vision and speech algorithm parameters may be fine-tuned for optimal performance.”; also, para 67, “For instance, a computer vision and speech application that produced computer vision and speech algorithm output data comprising computer vision and speech results that exceeded a certain variance threshold (e.g., more than X percentage away from the GT of a simulated hardware configuration) away from the underlying GT may be used by an artificial intelligence (AI) application running in the cloud environment 200 to optimize the deficient computer vision and speech application by running more testing against other synthetic scenes, motions, and hardware configurations until the computer vision and speech application performs within the variance threshold.”; also, para 82, “In step 810 a computer vision and speech algorithm, one of some number N, out of a set of algorithms is applied to the generated synthetic data, to obtain results 812 for the combination of device specification K, scene and EOC L, motion vector M, and algorithm N.”; also, para 63, “In some embodiments, motion orchestrator module 404 permits users of computer vision and speech design service 400 to model motion that is relevant for testing computer vision and speech sensor platforms and algorithms by expressing targeted motion profiles.”).
Regarding claim 7, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 1, wherein Ebstyne further discloses the retrained computer vision algorithm is configured for one of a plurality of virtual environments (para 78, “In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 67, “For instance, a computer vision and speech application that produced computer vision and speech algorithm output data comprising computer vision and speech results that exceeded a certain variance threshold (e.g., more than X percentage away from the GT of a simulated hardware configuration) away from the underlying GT may be used by an artificial intelligence (AI) application running in the cloud environment 200 to optimize the deficient computer vision and speech application by running more testing against other synthetic scenes, motions, and hardware configurations until the computer vision and speech application performs within the variance threshold.”; also, para 83, “Step 814a iterates on various different candidate computer vision and speech algorithms; step 8144 iterates on various different motion profiles; step 814c iterates on various different environments and EOCs; and step 814d iterates on various different candidate hardware configurations.”; also, para 75, “The illustrated environment 600 thus has the ability to conduct tests with multiple scenarios: (1) a single hardware configuration or multiple hardware configurations, each with (2) a single scene or multiple scenes; each with (3) a single motion profile or multiple motion profiles; each with (4) a single computer vision and speech algorithm or multiple different computer vision and speech algorithms.”).
Regarding claim 8, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 7, wherein Ebstyne further comprising: providing the retrained computer vision algorithm to the first device (para 78, “In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 56, “For example, the applications may represent downloaded client-side applications that correspond to server-side services executing in a cloud.”), wherein the first device is located in a third physical environment, the third physical environment corresponding to one of the plurality of virtual environments (para 31, “Just as a real-world camera may be moved around a room by users, capturing video of a room to generate data for a synthesized version of that room, a synthetic camera placed virtually into a synthetic scene can generate an equivalent data set.”; also, para 64, “Additionally or alternatively, environment orchestrator 406 defines the dimensions, objects, lighting, spacing, or other attributes of a room in scene and the contents therein.”).
Regarding claim 9, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 2, wherein Ebstyne further comprising: second sensor data from the second device in the second physical environment (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”); and determining the trajectory of the second device based on the second sensor data from the second device in the second physical environment (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs.”; also, para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”). Ebstyne does not disclose accessing first sensor data from the first device in the first physical environment; determining the gait pattern of the first user based on the first sensor data from the first device in the first physical environment.
However, in a similar field of endeavor, Trigoni disclose accessing first sensor data from the first device in the first physical environment (para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”; also, para 99, “However, tracking device orientation remains a major issue with low-cost inertial measurement unit (IMU) sensors embedded in mobile devices that are not constrained to be held in a certain way, which IMU sensors include a gyro and a magnetometer for the purpose.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”); determining the gait pattern of the first user based on the first sensor data from the first device in the first physical environment (para 130, ‘This embodiment uses a simple step length model [35], which uses accelerometer data to detect the step frequency (denoted with ft at time t) which is later used to estimate step length lt as follows:”; also, para 131, “where h is the pedestrian height, α is the step frequency coefficient, β is the step constant given pedestrian height, and γ is the step constant.”; also, para 154, “Different step constant parameters were learned for different pedestrians.”; also, para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne, in which inertial measurement unit data and six degree of freedom device sensor data are captured from a real-world device carried by a user walking around a real-world scene and a motion orchestrator determines the trajectory and orientation of the device under test from that capture, with the features of Trigoni's invention of accessing the accelerometer, magnetometer and gyroscope of a carried smart phone and determining from that accelerometer data a per-individual step length characterization. The combination would have been obvious because Ebstyne already performs the access step for both devices and already treats the captured sensor data as the source of the simulated trajectory, so the only thing added is the second use Trigoni makes of the same kind of data, and Trigoni states that the step constants so determined are very different for different pedestrians. A person of ordinary skill would have derived the first user's walking characterization from the first device's own sensor stream, as Trigoni does, rather than acquiring it by a separate modality, with the predictable result of a single consistent two-channel pipeline in which each device's sensor data serves the quantity that device is responsible for.
Regarding claim 10, Ebstyne as modified by Trigoni and Kurz discloses the method of claim 9, second sensor data comprise 6DOF poses of the second device (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT). “; also, para 58, “The combination of 3D position and 3D rotation may be referred to as six degrees-of-freedom (6DoF), and a combination of 3D accelerometer and 3D gyroscope data may permit 6DoF measurements.”) in the second physical environment (para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs”). Ebstyne does not disclose the first sensor data comprise 6DOF poses of the first device over time in the first physical environment.
However, in a similar field of endeavor disclose the first sensor data comprise 6DOF poses of the first device (para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”; also, para 85, “The output of the R-PDR layer 10, a trajectory for the user, in particular in this embodiment it comprises a distance value and heading value for each step taken by the user of the device.”; also, para 107, “As noted above, the gravity vector estimates are then fed into the Kalman filter as additional observations (along with magnetometer measurements) to estimate the orientation of the device.”) over time in the first physical environment (para 142, ‘We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne, whose capture supplies the second device's position and orientation, with the features of Trigoni's invention of a carried smart phone whose accelerometer, magnetometer and gyroscope yield a per-step trajectory and an estimated device orientation in an office, a museum and a market. The combination would have been obvious because the claim assigns the first device its own sensor stream and Ebstyne performs only one capture, so a person of ordinary skill needed a separate first-device acquisition, which Trigoni supplies in the same modality Ebstyne already uses. The predictable result is a two-channel pipeline in which the first device's own inertial and trajectory data supply the first sensor data in the first physical environment.
Kurz discloses over time (sec 3.1, “The resulting camera motion is a regularly-sampled time series of camera positions and orientations.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Trigoni, in which each device's position and orientation are captured, with the features of Kurz's invention of representing a camera's motion as a regularly-sampled time series of positions and orientations. The combination would have been obvious because Ebstyne and Trigoni each produce a device's position and orientation but neither states the sampling in time, and Kurz supplies exactly that representation for the same quantity. A person of ordinary skill would have kept the captured positions and orientations as a time-ordered series, with the predictable result that the sensor data comprise poses of the device over time.
Regarding claim 11, Ebstyne discloses a computing apparatus comprising (para 52, “Computing device 302 represents any device executing instructions (e.g., as application programs, operating system functionality, or both) to implement the operations and functionality as described herein.”): a processor (para 53, “In some examples, computing device 302 has at least one processor 304, a memory area 306, and at least one user interface.”; also, para 53, “Processor 304 includes any quantity of processing units, and is programmed to execute computer-executable instructions for implementing aspects of the disclosure.”); and a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations comprising (para 56, “Memory area 308 stores, among other data, one or more applications or algorithms 308 that include both data and executable instructions 310. The applications, when executed by the processor, operate to perform functionality on the computing device.”; also, para 55, “Computing device 302 further has one or more computer readable media such as the memory area 306.”; also, para 36, “Computer-storage memory 112 may be used to store and access instructions configured to carry out the various operations disclosed herein.”): generating simulated ground truth data (para 76, “Evaluator 636 may be used for comparing the calculated computer vision and speech data with ground truth data.”; also, para 76, “Because environment orchestrator 406 and motion orchestrator 404 produce the output for simulator 624 to use, the information about where the synthetic DUT is within the synthetic environment is available for use in the evaluation process.”; also, para 76, “This known data, actual position of DUT and objects) [sic] is collectively known as the ground truth (GT) 638.”; also, para 80, “Generating synthetic experiment data comprises simulating, for one or more hardware configurations sensor data that can be supplied to a computer vision and speech algorithm.”) based on a modified trajectory (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”) of a second device (para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs.”; also, para 52, “That is, the computing device 302 may represent a real-world device that is designed using an end-to-end computer vision and speech design service, or may represent a synthetic version used as a test candidate for data generation.”; also, para 52, “In some examples, a mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, wearable device, head mounted display (HMD) and/or portable media player.”) of a second user (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 31, “Just as a real-world camera may be moved around a room by users, capturing video of a room to generate data for a synthesized version of that room, a synthetic camera placed virtually into a synthetic scene can generate an equivalent data set.”) in a second physical environment (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 20, “Setting up an experimentation workflow for a manual build-test-repeat type process typically requires dependence on manually collecting and labelling large amounts of data that must be collected by sending testers with prototype devices into real-world environments to collect sensor data streams.”), retraining a computer vision algorithm with the simulated ground truth data 9para 78, “In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 78, “For these lesser-accurate computer vision and speech applications, disparity data of the simulated virtual hardware configuration compared with its GT data 702 may be fed back to the algorithm application 710 to improve the performance of such computer vision and speech applications.”). Ebstyne does not disclose determining a gait pattern of a first user of a first device in a first physical environment, the gait pattern determined by a machine-learning model trained by an unsupervised training algorithm using inertial measurement unit data from an inertial sensor of the first device and poses of the first device over time in the first physical environment; the modified trajectory adapting poses of the second device over time to the gait pattern of the first user.
However, in a similar field of endeavor, Trigoni disclose determining a gait pattern of a first user of a first device in a first physical environment (para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”; also, para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”), the gait pattern determined by a machine-learning model trained by an unsupervised training algorithm (para 138, “To address this issue, to [sic] the present embodiment uses an unsupervised approach to learning R-PDR parameters.”; also, para 132, “This is referred to here as unsupervised lifelong learning approach as LL-Tracker, and describe in detail later below.”; also , 139, “The solution to this optimization can be obtained with the expectation maximization (EM) approach.”; also, para 130, “This embodiment uses a simple step length model [35], which uses accelerometer data to detect the step frequency (denoted with ft at time t) which is later used to estimate step length lt as follows:”; also, para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 139, “Therefore, the hard EM approach, also known as Viterbi training is employed to solve the optimization problem [39].”) using inertial measurement unit data from an inertial sensor of the first device (para 99, “However, tracking device orientation remains a major issue with low-cost inertial measurement unit (IMU) sensors embedded in mobile devices that are not constrained to be held in a certain way, which IMU sensors include a gyro and a magnetometer for the purpose.”; also, para 82, “The accelerometer, magnetometer and gyroscope are the main sensors but others could be taken into account by the system to provide position, acceleration and orientation signals, such as a barometer, light sensors and so on.”) and poses of the first device over time in the first physical environment (para 85, “The output of the R-PDR layer 10, a trajectory for the user, in particular in this embodiment it comprises a distance value and heading value for each step taken by the user of the device.”; also, para 107, “noted above, the gravity vector estimates are then fed into the Kalman filter as additional observations (along with magnetometer measurements) to estimate the orientation of the device.”; also, para 139, “(x) is the matched trajectory and Z(x) is the raw trajectory fed from R-PDR to map matching.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”); the gait pattern of the first user (para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 154, “Different step constant parameters were learned for different pedestrians.”; also, para 132, “The accuracy of this model is not satisfactory by itself because step length parameters vary across different users and environments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ebstyne's invention of a computing apparatus having a processor and a memory storing executable instructions that generate ground truth data and simulated sensor data, simulate the trajectory and orientation of a synthetic device under test from motion recorded from actual real-world devices captured by a user walking around a real-world scene, and feed the disparity between the simulated data and the ground truth data back to improve the computer vision applications, with the features of Trigoni's invention of learning, by an unsupervised approach using expectation maximization, a per-individual step constant from the inertial measurement unit sensors embedded in a carried smart phone together with that device's trajectory and estimated orientation. The combination would have been obvious because both references run their computations on a processor operating on sensor data captured at a carried device, and Ebstyne already collects the two streams Trigoni learns from without ever characterizing the walking of the person who carried the device. Trigoni states that step length parameters vary across different users and that different step constant parameters were learned for different pedestrians, so a person of ordinary skill seeking a per-user description of the captured walk would have added Trigoni's unsupervised learning to the instructions the apparatus already executes, with the predictable result that the same apparatus produces both the simulated trajectory and the walking characterization. The characterization so learned belongs to the particular first user whose device supplied the data, because Trigoni states that different step constant parameters were learned for different pedestrians and that step length parameters vary across different users, so it is that user's gait pattern to which the second device's poses are adapted.
Kurz discloses the modified trajectory adapting poses of the second device over time (sec 3.1 “The resulting camera motion is a regularly-sampled time series of camera positions and orientations. Such a representation has six degrees of freedom per frame, where position is stored as an Euclidean 3-vector x and orientation as a unit quaternion q.”; also, sec 1, “Therefore, in this paper we present an approach to transfer camera shake from real videos onto an artificially designed camera motion path in a virtual scene.”; also, sec 4, “First, the user selects a simple walking motion from a certain movie as style. Our system transfers this walking style to a different camera animation.”; also, sec 3.1, “From this data, we reconstruct the three-dimensional camera motion paths originally used in the shooting of the footage.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Trigoni, in which a processor executes instructions that learn a per-user walking characterization without labels from a carried device's inertial sensors and trajectory while a motion orchestrator simulates the trajectory and orientation of a device under test, with the features of Kurz's invention of transferring a walking motion recovered from a real recorded camera path onto a different camera path represented as six degrees of freedom of position and orientation per frame. The combination would have been obvious because the base combination leaves the walking characterization and the simulated trajectory as two unconnected outputs of the same apparatus, and Kurz supplies the operation that joins them by imposing a walking motion measured from one camera path onto another.
Regarding claim 12, Ebstyne as modified by Trigoni and Kurz discloses the computing apparatus of claim 11, wherein Ebstyne further discloses the operations further comprise: identifying a trajectory (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”) of the second device (para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs.; also, para 52, “That is, the computing device 302 may represent a real-world device that is designed using an end-to-end computer vision and speech design service, or may represent a synthetic version used as a test candidate for data generation.”; also, para 52, “In some examples, a mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, wearable device, head mounted display (HMD) and/or portable media player.”) of the second user (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 31, “Just as a real-world camera may be moved around a room by users, capturing video of a room to generate data for a synthesized version of that room, a synthetic camera placed virtually into a synthetic scene can generate an equivalent data set.”) in the second physical environment (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 20, “Setting up an experimentation workflow for a manual build-test-repeat type process typically requires dependence on manually collecting and labelling large amounts of data that must be collected by sending testers with prototype devices into real-world environments to collect sensor data streams.”); applying the modified trajectory to a plurality of virtual environments (para 64, “Environment orchestrator 406 is used for simulating one or more virtual environments. In some embodiments, environment orchestrator 406 permits users to manipulate synthetic environments, such as light settings and the state of certain objects, such as doors.”; also, para 63, “Motion orchestrator 404 may be used for simulating motion of the one or more simulated hardware configurations within one or more virtual environments. Examples may include creating instances of multiple random walks through a virtual scene or room, having various durations, speeds, and motion pathways.”). Ebstyne does not disclose determining the modified trajectory of the second device based on the gait pattern of the first user.
However, in a similar field of endeavor, Kurz discloses determining the modified trajectory of the second device (sec 4, “First, the user selects a simple walking motion from a certain movie as style. Our system transfers this walking style to a different camera animation. “; also, sec 3.1, “The resulting camera motion is a regularly-sampled time series of camera positions and orientations. Such a representation has six degrees of freedom per frame, where position is stored as an Euclidean 3-vector x and orientation as a unit quaternion q. “; also, abstract, “Consequently, an arbitrary virtual base motion, defined in any conventional animation package, can be automatically modified according to a user-selected style.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Trigoni, in which the apparatus simulates the trajectory and orientation of a device under test through one or more virtual environments and learns a per-user walking characterization from a carried device's inertial and trajectory data, with the features of Kurz's invention of transferring a walking motion selected from a recorded camera path onto a different camera animation. The combination would have been obvious because the motion orchestrator already takes a motion profile as input and already drives the device under test through several virtual environments, so what a person of ordinary skill needed was a way to express the learned walking characterization as that profile, and Kurz provides a worked technique for doing so.
Trigoni discloses based on the gait pattern of the first user (para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 154, “Different step constant parameters were learned for different pedestrians.”; also, para 132, “The accuracy of this model is not satisfactory by itself because step length parameters vary across different users and environments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Kurz, in which the apparatus imposes a walking characterization on the trajectory it simulates, with the features of Trigoni's invention of learning a step constant separately for each individual pedestrian from that pedestrian's own carried device. The combination would have been obvious because the characterization the apparatus imposes has to be measured from something, and Trigoni already supplies it as a per-person quantity, stating that different step constant parameters were learned for different pedestrians and that step length parameters vary across different users and environments. A person of ordinary skill would therefore have conditioned the determination on the walking characterization learned from the first user's own device rather than on a generic walk, with the predictable result that the modified trajectory reflects that individual's walk.
Regarding claim 13, Ebstyne as modified by Trigoni and Kurz discloses the computing apparatus of claim 12, wherein Ebstyne further discloses the first device comprises a first visual tracking device (para 52, “In some examples, a mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, wearable device, head mounted display (HMD) and/or portable media player.”; also, para 58, “As illustrated, computing device 302 further includes a camera 330 (though other types of sensors may be used), which may represent a single camera, a stereo camera set, a set of differently-facing cameras, or another configuration.”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”), wherein the second device comprises a second visual tracking device (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs. Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”), wherein the trajectory is based on poses of the second visual tracking device over time (para 58, “The combination of 3D position and 3D rotation may be referred to as six degrees-of-freedom (6DoF), and a combination of 3D accelerometer and 3D gyroscope data may permit 6DoF measurements.”; also, para 81, “In step 806, some number, M, of different device positions and orientations are generated.”; also, para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”).
Regarding claim 14, Ebstyne as modified by Trigoni and Kurz discloses the computing apparatus of claim 11, wherein Ebstyne further discloses the modified trajectory is applied to a plurality of virtual environments (para 63, “Motion orchestrator 404 may be used for simulating motion of the one or more simulated hardware configurations within one or more virtual environments. Examples may include creating instances of multiple random walks through a virtual scene or room, having various durations, speeds, and motion pathways.”; also, para 64, “Environment orchestrator 406 is used for simulating one or more virtual environments.”).
Regarding claim 15, Ebstyne as modified by Trigoni and Kurz discloses the computing apparatus of claim 11, wherein Ebstyne further discloses the retrained computer vision algorithm is configured for a motion pattern of the first user (para 78, ‘In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 67, “For instance, a computer vision and speech application that produced computer vision and speech algorithm output data comprising computer vision and speech results that exceeded a certain variance threshold (e.g., more than X percentage away from the GT of a simulated hardware configuration) away from the underlying GT may be used by an artificial intelligence (AI) application running in the cloud environment 200 to optimize the deficient computer vision and speech application by running more testing against other synthetic scenes, motions, and hardware configurations until the computer vision and speech application performs within the variance threshold.”; also, para 83, “Step 814a iterates on various different candidate computer vision and speech algorithms; step 8144 iterates on various different motion profiles; step 814c iterates on various different environments and EOCs; and step 814d iterates on various different candidate hardware configurations.”; also, para 63, “In some embodiments, motion orchestrator module 404 permits users of computer vision and speech design service 400 to model motion that is relevant for testing computer vision and speech sensor platforms and algorithms by expressing targeted motion profiles.”).
Regarding claim 16, Ebstyne as modified by Trigoni and Kurz discloses the computing apparatus of claim 15, wherein Ebstyne further discloses the operations further comprise: providing the retrained computer vision algorithm to the first device (para 78, ‘In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 56, “For example, the applications may represent downloaded client-side applications that correspond to server-side services executing in a cloud.”; also, para 56, “Exemplary applications include computer vision and speech applications having computer vision and speech algorithms for identifying the coordinates of computing device 302.”), wherein the first device is re-configured for the motion pattern of the first user based on the retrained computer vision algorithm (para 84, “Experimental data is evaluated in step 816, for example by comparing computer vision and speech algorithm output data with GT data, and a recommendation step 818 may provide a recommended best performing configuration or data from which a designer may select an adequately performing configuration.”; also, para 84, “In this manner hardware configurations may be selected from a set of candidate hardware configurations, computer vision and speech algorithms may be selected from a set of candidate algorithms, or computer vision and speech algorithm parameters may be fine-tuned for optimal performance.”; also, para 67, “For instance, a computer vision and speech application that produced computer vision and speech algorithm output data comprising computer vision and speech results that exceeded a certain variance threshold (e.g., more than X percentage away from the GT of a simulated hardware configuration) away from the underlying GT may be used by an artificial intelligence (AI) application running in the cloud environment 200 to optimize the deficient computer vision and speech application by running more testing against other synthetic scenes, motions, and hardware configurations until the computer vision and speech application performs within the variance threshold.”; also, para 82, “In step 810 a computer vision and speech algorithm, one of some number N, out of a set of algorithms is applied to the generated synthetic data, to obtain results 812 for the combination of device specification K, scene and EOC L, motion vector M, and algorithm N.”; also, para 63, “In some embodiments, motion orchestrator module 404 permits users of computer vision and speech design service 400 to model motion that is relevant for testing computer vision and speech sensor platforms and algorithms by expressing targeted motion profiles.”).
Regarding claim 17, Ebstyne as modified by Trigoni and Kurz discloses computing apparatus of claim 11, wherein Ebstyne further discloses the retrained computer vision algorithm is configured for one of a plurality of virtual environments (para 78, “In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 67, “For instance, a computer vision and speech application that produced computer vision and speech algorithm output data comprising computer vision and speech results that exceeded a certain variance threshold (e.g., more than X percentage away from the GT of a simulated hardware configuration) away from the underlying GT may be used by an artificial intelligence (AI) application running in the cloud environment 200 to optimize the deficient computer vision and speech application by running more testing against other synthetic scenes, motions, and hardware configurations until the computer vision and speech application performs within the variance threshold.”; also, para 83, “Step 814a iterates on various different candidate computer vision and speech algorithms; step 8144 iterates on various different motion profiles; step 814c iterates on various different environments and EOCs; and step 814d iterates on various different candidate hardware configurations.”; also, para 75, “The illustrated environment 600 thus has the ability to conduct tests with multiple scenarios: (1) a single hardware configuration or multiple hardware configurations, each with (2) a single scene or multiple scenes; each with (3) a single motion profile or multiple motion profiles; each with (4) a single computer vision and speech algorithm or multiple different computer vision and speech algorithms.”).
Regarding claim 18, Ebstyne as modified by Trigoni and Kurz discloses the computing apparatus of claim 17, further comprising: providing the retrained computer vision algorithm to the first device (para 78, ‘In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 56, “For example, the applications may represent downloaded client-side applications that correspond to server-side services executing in a cloud.”), wherein the first device is located in a third physical environment, the third physical environment corresponding to one of the plurality of virtual environments (para 31, ‘Just as a real-world camera may be moved around a room by users, capturing video of a room to generate data for a synthesized version of that room, a synthetic camera placed virtually into a synthetic scene can generate an equivalent data set.”; also, para 64, “Additionally or alternatively, environment orchestrator 406 defines the dimensions, objects, lighting, spacing, or other attributes of a room in scene and the contents therein.”).
Regarding claim 19, Ebstyne as modified by Trigoni and Kurz discloses the computing apparatus of claim 12, wherein Ebstyne further comprising wherein the operations further comprise: accessing second sensor data from the second device in the second physical environment (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”); and determining the trajectory of the second device based on the second sensor data from the second device in the second physical environment (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs.”), second sensor data comprise 6DOF poses of the first second device second physical environment (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 58, “The combination of 3D position and 3D rotation may be referred to as six degrees-of-freedom (6DoF), and a combination of 3D accelerometer and 3D gyroscope data may permit 6DoF measurements.”). Ebstyne does not disclose accessing first sensor data from the first device in the first physical environment; determining the gait pattern of the first user based on the first sensor data from the first device in the first physical environment; wherein the first sensor data comprise 6DOF poses of the first device over time in the first physical environment; over time.
However, in a similar field of endeavor, Trigoni discloses accessing first sensor data from the first device in the first physical environment (para 82, ‘A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”; also, para 99, “However, tracking device orientation remains a major issue with low-cost inertial measurement unit (IMU) sensors embedded in mobile devices that are not constrained to be held in a certain way, which IMU sensors include a gyro and a magnetometer for the purpose.”; also, para 142, ‘We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”); determining the gait pattern of the first user based on the first sensor data from the first device in the first physical environment (para 130, “This embodiment uses a simple step length model [35], which uses accelerometer data to detect the step frequency (denoted with ft at time t) which is later used to estimate step length lt as follows:”; also, para 131, ‘where h is the pedestrian height, α is the step frequency coefficient, β is the step constant given pedestrian height, and γ is the step constant.”; also, para 154, “Different step constant parameters were learned for different pedestrians.”; also, para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”); wherein the first sensor data comprise 6DOF poses of the first device (para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”; also, para 85, “The output of the R-PDR layer 10, a trajectory for the user, in particular in this embodiment it comprises a distance value and heading value for each step taken by the user of the device.”; also, para 107, “As noted above, the gravity vector estimates are then fed into the Kalman filter as additional observations (along with magnetometer measurements) to estimate the orientation of the device.”) in the first physical environment (para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne, whose capture supplies the second device's position and orientation, with the features of Trigoni's invention of a carried smart phone whose accelerometer, magnetometer and gyroscope yield a per-step trajectory and an estimated device orientation in an office, a museum and a market. The combination would have been obvious because the claim assigns the first device its own sensor stream and Ebstyne performs only one capture, so a person of ordinary skill needed a separate first-device acquisition, which Trigoni supplies in the same modality Ebstyne already uses. The predictable result is a two-channel pipeline in which the first device's own inertial and trajectory data supply the first sensor data in the first physical environment.
Kurz discloses over time (sec 3.1, ‘The resulting camera motion is a regularly-sampled time series of camera positions and orientations.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Trigoni, in which each device's position and orientation are captured, with the features of Kurz's invention of representing a camera's motion as a regularly-sampled time series of positions and orientations. The combination would have been obvious because Ebstyne and Trigoni each produce a device's position and orientation but neither states the sampling in time, and Kurz supplies exactly that representation for the same quantity. A person of ordinary skill would have kept the captured positions and orientations as a time-ordered series, with the predictable result that the sensor data comprise poses of the device over time.
Regarding claim 20, Ebstyne discloses a non-transitory computer-readable storage medium (para 96, “Computer storage media are tangible and mutually exclusive to communication media. Computer storage media are implemented in hardware and exclude carrier waves and propagated signals. Computer storage media for purposes of this disclosure are not signals per se.”), the computer- readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising (para 56, “Memory area 308 stores, among other data, one or more applications or algorithms 308 that include both data and executable instructions 310. The applications, when executed by the processor, operate to perform functionality on the computing device.”; also, para 95, “Examples of the disclosure may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices in software, firmware, hardware, or a combination thereof.”; also, para 93, “Still other examples are directed to one or more computer storage devices having computer-executable instructions stored thereon for developing a computer vision or speech solution, which, on execution by a computer, cause the computer to perform operations, the instructions comprising: a sensor platform simulator component for simulating one or more hardware configurations comprising the one or more virtual sensors; an environment orchestrator component for simulating one or more virtual environments; a motion orchestrator component for simulating motion of the one or more simulated hardware configurations within the one or more virtual environments; an experiment generator component for generating synthetic experiment data for a plurality of candidate computer vision or speech solutions having differing hardware configurations or computer vision or speech algorithm parameters; and an experiment runner component for iterating the experiment generator to generate the synthetic experiment data for one or more combinations of hardware configurations, virtual environment, motion and computer vision or speech algorithms.”): generating simulated ground truth data (para 76, “Evaluator 636 may be used for comparing the calculated computer vision and speech data with ground truth data.”; also, para 76, “Because environment orchestrator 406 and motion orchestrator 404 produce the output for simulator 624 to use, the information about where the synthetic DUT is within the synthetic environment is available for use in the evaluation process.”; also, para 76, “This known data, actual position of DUT and objects) [sic] is collectively known as the ground truth (GT) 638.”; also, para 80, “Generating synthetic experiment data comprises simulating, for one or more hardware configurations sensor data that can be supplied to a computer vision and speech algorithm.”) based on a modified trajectory (para 73, “Motion orchestrator 404 simulates the trajectory and orientation of the synthetic device under test (DUT).”; also, para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”) of a second device (para 73, “The motion may be modeled or recorded from actual real-world devices, to provide typical movement scenarios for VR devices, including HMDs.”; also, para 52, “That is, the computing device 302 may represent a real-world device that is designed using an end-to-end computer vision and speech design service, or may represent a synthetic version used as a test candidate for data generation.”; also, para 52, “In some examples, a mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, wearable device, head mounted display (HMD) and/or portable media player.”) of a second user (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 31, ‘Just as a real-world camera may be moved around a room by users, capturing video of a room to generate data for a synthesized version of that room, a synthetic camera placed virtually into a synthetic scene can generate an equivalent data set.”) in a second physical environment (para 73, “Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest.”; also, para 20, “Setting up an experimentation workflow for a manual build-test-repeat type process typically requires dependence on manually collecting and labelling large amounts of data that must be collected by sending testers with prototype devices into real-world environments to collect sensor data streams.”), retraining a computer vision algorithm with the simulated ground truth data (para 78, “In some embodiments, the algorithm application 710 uses AI processing or machine-learning to improve the computer vision and speech applications in the library 712 based on—or triggered by—the evaluation results 720.”; also, para 78, “For these lesser-accurate computer vision and speech applications, disparity data of the simulated virtual hardware configuration compared with its GT data 702 may be fed back to the algorithm application 710 to improve the performance of such computer vision and speech applications.”). Ebstyne does not disclose determining a gait pattern of a first user of a first device in a first physical environment, the gait pattern determined by a machine-learning model trained by an unsupervised training algorithm using inertial measurement unit data from an inertial sensor of the first device and poses of the first device over time in the first physical environment; the modified trajectory adapting poses of the second device over time to the gait pattern of the first user.
However, in a similar field of endeavor, Trigoni discloses determining a gait pattern of a first user of a first device in a first physical environment (para 142, ‘In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”; also, para 82, “A smart phone 100 having sensors, such as an accelerometer, magnetometer, gyroscope, etc. is provided with software, the modules of which are shown in the rest of the Figure.”), the gait pattern determined by a machine-learning model trained by an unsupervised training algorithm (para 138, “To address this issue, to [sic] the present embodiment uses an unsupervised approach to learning R-PDR parameters.”; also, para 132, “This is referred to here as unsupervised lifelong learning approach as LL-Tracker, and describe in detail later below.”; also, 139, ‘The solution to this optimization can be obtained with the expectation maximization (EM) approach.”; also, para 130, “This embodiment uses a simple step length model [35], which uses accelerometer data to detect the step frequency (denoted with ft at time t) which is later used to estimate step length lt as follows:”; also, para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 139, “Therefore, the hard EM approach, also known as Viterbi training is employed to solve the optimization problem [39].”) using inertial measurement unit data from an inertial sensor of the first device (para 99, “However, tracking device orientation remains a major issue with low-cost inertial measurement unit (IMU) sensors embedded in mobile devices that are not constrained to be held in a certain way, which IMU sensors include a gyro and a magnetometer for the purpose.”; also, para 82, “The accelerometer, magnetometer and gyroscope are the main sensors but others could be taken into account by the system to provide position, acceleration and orientation signals, such as a barometer, light sensors and so on.”) and poses of the first device over time in the first physical environment (para 85, “The output of the R-PDR layer 10, a trajectory for the user, in particular in this embodiment it comprises a distance value and heading value for each step taken by the user of the device.”; also, para 107, “As noted above, the gravity vector estimates are then fed into the Kalman filter as additional observations (along with magnetometer measurements) to estimate the orientation of the device.”; also, para 139, “S(x) is the matched trajectory and Z(x) is the raw trajectory fed from R-PDR to map matching.”; also, para 142, “We have conducted experiments in an office environment, a museum environment, and a market to show the effectiveness of the feedback loop on step length estimation.”); the gait pattern of the first user (para 142, “In particular, the method of this embodiment was used to learn the step constant γ (Eqn. (5)) for different individuals because 1) the average step length plays a crucial role in the tracking accuracy due to the high consistency of step length in human walking patterns; and 2) the parameters α and β are very similar for different individuals in our experiments.”; also, para 154, “Different step constant parameters were learned for different pedestrians.”; also, para 132, “The accuracy of this model is not satisfactory by itself because step length parameters vary across different users and environments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ebstyne's invention of a tangible computer storage medium that is not a signal per se and that stores executable instructions which generate ground truth data and simulated sensor data, simulate the trajectory and orientation of a synthetic device under test from motion recorded from actual real-world devices captured by a user walking around a real-world scene, and feed the disparity between the simulated data and the ground truth data back to improve the computer vision applications, with the features of Trigoni's invention of learning, by an unsupervised approach using expectation maximization, a per-individual step constant from the inertial measurement unit sensors embedded in a carried smart phone together with that device's trajectory and estimated orientation. The combination would have been obvious because Trigoni's own disclosure is software distributed to a smart phone, so its routine is of the same kind as the instructions Ebstyne already stores, and Ebstyne already captures the two data streams Trigoni learns from while stopping short of characterizing the walk of the person who carried the device. Trigoni states that step length parameters vary across different users and that different step constant parameters were learned for different pedestrians, so a person of ordinary skill preparing a storage medium for this pipeline would have included instructions for both routines, with the predictable result that one medium carries both the simulation and the per-user walking characterization it is to be conditioned on. The characterization so learned belongs to the particular first user whose device supplied the data, because Trigoni states that different step constant parameters were learned for different pedestrians and that step length parameters vary across different users, so it is that user's gait pattern to which the second device's poses are adapted.
Kurz discloses the modified trajectory adapting poses of the second device over time (sec 3.1 “The resulting camera motion is a regularly-sampled time series of camera positions and orientations. Such a representation has six degrees of freedom per frame, where position is stored as an Euclidean 3-vector x and orientation as a unit quaternion q.”; also, sec 1, “Therefore, in this paper we present an approach to transfer camera shake from real videos onto an artificially designed camera motion path in a virtual scene.”; also, sec 4, “First, the user selects a simple walking motion from a certain movie as style. Our system transfers this walking style to a different camera animation.”; also, sec 3.1, “From this data, we reconstruct the three-dimensional camera motion paths originally used in the shooting of the footage.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Ebstyne in view of Trigoni, in which a storage medium carries instructions that learn a per-user walking characterization without labels from a carried device's inertial sensors and trajectory and that simulate the trajectory and orientation of a device under test, with the features of Kurz's invention of transferring a walking motion recovered from a real recorded camera path onto a different camera path represented as six degrees of freedom of position and orientation per frame. The combination would have been obvious because the instructions already on the medium produce the walking characterization and the simulated trajectory as two unconnected quantities, and Kurz supplies the routine that joins them by imposing a walking motion measured from one camera path onto another.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 11, and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
On pages 8 and 9 of the Applicant's Remarks, with respect to the rejection of claims 1, 11 and 20, the Applicant argues under heading A that "Sterling determines gait from a different data source than the amended claims recite," that "Sterling does not use either input to determine gait," and that "A pose stream of Sterling's client device would describe the motion of the camera, not the gait of the imaged subject." These arguments are persuasive. Sterling determines gait from two-dimensional image or video data of the subject's body and performs anatomical pose estimation of the imaged subject rather than deriving a gait pattern from an inertial sensor of the device and the poses of that device over time, and the rejection over Sterling has therefore been withdrawn. However, upon further consideration and as necessitated by Applicant's amendment, a new ground of rejection under 35 U.S.C. 103 over Ebstyne in view of Trigoni, further in view of Kurz is made as set forth above.
On pages 9 and 10 of the Applicant's Remarks, with respect to the rejection of claims 1, 11 and 20, the Applicant argues under heading B that "Sterling's machine-learning routines are trained on labeled data" and that "Sterling does not disclose training without labels," and further that "Ebstyne does not disclose an unsupervised training algorithm operating on untagged inertial and pose data to produce a model that identifies a gait pattern of a user." These arguments are persuasive as to Sterling and Ebstyne and that rejection has been withdrawn. As shown in the citations, Trigoni discloses that "To address this issue, to [sic] the present embodiment uses an unsupervised approach to learning R-PDR parameters" (Trigoni, paragraph [0138]) and that "This is referred to here as unsupervised lifelong learning approach as LL-Tracker, and describe in detail later below" (Trigoni, paragraph [0132]), and identifies the algorithm by which that learning is performed, stating that "The solution to this optimization can be obtained with the expectation maximization (EM) approach" (Trigoni, paragraph [0139]). What is learned is a characterization of an individual's walking, because Trigoni states that "Different step constant parameters were learned for different pedestrians" (Trigoni, paragraph [0154]), and the data the learning operates on is the device's own inertial and trajectory data, because Trigoni states that "The accelerometer, magnetometer and gyroscope are the main sensors but others could be taken into account by the system to provide position, acceleration and orientation signals, such as a barometer, light sensors and so on" (Trigoni, paragraph [0082]) and that "S(x) is the matched trajectory and Z(x) is the raw trajectory fed from R-PDR to map matching" (Trigoni, paragraph [0139]). A per-individual step constant learned without labels from the carried device's inertial data and its trajectory is a data characterization of the manner in which that person walks.
On page 10 of the Applicant's Remarks, with respect to the rejection of claims 1, 11 and 20, the Applicant argues under heading C that "Reconfiguring Sterling to derive the gait pattern from inertial measurement unit data and six-degree-of-freedom poses of a device carried by the subject substitutes the sensing modality that Sterling identifies as the deficiency its framework addresses" and that "Sterling provides no direction toward that substitution." This argument is persuasive and the rejection resting on that substitution has been withdrawn. The rejection set forth above requires no such substitution, because Trigoni derives its walking characterization from the carried device's inertial sensors and trajectory in the first instance and is not reconfigured from a video modality.
On page 11 of the Applicant's Remarks, with respect to the rejection of claims 2 and 12, the Applicant argues under heading D that "Sterling operates on one subject and one gait cycle" and that "Sterling contains no second subject, no second device trajectory, and no application of one subject's gait to another subject's recorded motion." This argument is persuasive as to Sterling and that rejection has been withdrawn. As shown in the citations, Kurz supplies the operation the Applicant identifies as absent, disclosing that "First, the user selects a simple walking motion from a certain movie as style. Our system transfers this walking style to a different camera animation" (Kurz, section 4), and disclosing that what is transferred is measured from a real recorded camera path, because "From this data, we reconstruct the three-dimensional camera-motion paths originally used in the shooting of the footage" (Kurz, section 3.1) and "For the data acquisition we used selected videos of varying camera motions from different sources, including casual home video, feature films and TV series" (Kurz, section 3.1). The quantity Kurz modifies is a sequence of camera positions and orientations rather than a subject's own movement, because "The resulting camera motion is a regularly-sampled time series of camera positions and orientations. Such a representation has six degrees of freedom per frame, where position is stored as an Euclidean 3-vector x and orientation as a unit quaternion q" (Kurz, section 3.1). Kurz therefore teaches taking a walking motion measured from one camera's recorded path and imposing it on a second, different camera path whose every frame carries a position and an orientation, which is the transfer from one motion source to a separate second path that the Applicant states Sterling lacks. Kurz applies that transfer to a camera path rather than to a device, and the device whose poses are adapted is supplied by Ebstyne, whose motion orchestrator simulates the trajectory and orientation of the device under test from motion modeled or recorded from actual real-world devices. As shown in the citations, Trigoni supplies the further teaching that the walk so imposed is an individual's, disclosing that "Different step constant parameters were learned for different pedestrians" (Trigoni, paragraph [0154]) and that "The accuracy of this model is not satisfactory by itself because step length parameters vary across different users and environments" (Trigoni, paragraph [0132]), so the characterization being imposed is the one learned from the first user's own carried device rather than a generic walk.
On pages 11 and 12 of the Applicant's Remarks, with respect to the rejection of claims 1, 11 and 20, the Applicant argues under heading E that "Duration, speed, and motion pathway are parameters of a random walk" and that "Ebstyne does not disclose an input by which a motion profile is conditioned on a gait pattern of an identified individual, and does not disclose an interface for receiving such a parameter." This argument has been considered but is moot because it does not apply to the new combination of references being used in the current rejection. The quantity the combination imposes on the motion orchestrator's trajectory is a walking characterization measured from a particular person's own carried device, which Trigoni learns without labels from that device's inertial data and its trajectory and which Trigoni states is learned separately for different pedestrians, and the operation by which a measured walking motion is imposed on a separate sequence of camera positions and orientations over time is the transfer Kurz performs.
On page 12 of the Applicant's Remarks, with respect to the rejection of claim 19, the Applicant argues under heading F that "The noticed fact is directed to hardware integration" and requests that "To the extent the Official Notice is relied upon for any proposition beyond the availability of that hardware configuration, documentary evidence is requested in accordance with MPEP § 2144.03(C)." This argument is persuasive. The Official Notice taken in the prior action is withdrawn and is not relied upon in this action. As shown in the citations, the limitation is now supported by a citation of record, because Ebstyne discloses that "The combination of 3D position and 3D rotation may be referred to as six degrees-of-freedom (6DoF), and a combination of 3D accelerometer and 3D gyroscope data may permit 6DoF measurements" (Ebstyne, paragraph [0058]) and that "Both IMU data and data from other 6DoF device sensors may be captured, perhaps by a user walking around a real-world scene, that is later modeled, and pointing the cameras or other sensors at various points of interest" (Ebstyne, paragraph [0073]).
On page 12 of the Applicant's Remarks, with respect to the rejection of claims 3-10 and 13-19, the Applicant argues under heading G that each dependent claim "incorporates the limitations of the independent claim from which it depends and is allowable for at least the reasons set forth above, in addition to the further limitations each recites." This argument is not persuasive. The arguments directed to the independent claims are answered above, and each dependent claim is separately mapped in full in the rejection set forth above, with a quote placed under every limitation each recites.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Patent Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights