Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6-8, 12-14, 19 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baig (US 20200090344 A1) and Mei et al (CN115937451A), hereinafter Baig and Mei respectively.
Regarding claim 14, Baig teaches a device comprising: a non-transitory computer-readable storage medium;
"Advantages of such embodiments may include that the method is used on a readily available device including components that can be used to perform steps of the method." - Par 21 Baig
NOTE: See also HMD device 100, par 43
and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors,
Some embodiments provide a non-transitory storage medium (e.g. ROM 121, RAM 122) containing program instructions (i.e. software) that, when executed by a computer processor (e.g. CPU 140 or processing section 167), perform the methods described herein." - Par 110
cause the one or more processors to perform operations comprising: obtaining sensor data for a sequence of frames by the one or more sensors in a physical environment, wherein the physical environment includes an object;
"The method includes acquiring an earlier image and a later image of the environment from an image stream captured by the camera. The method further includes identifying later environment features located in the environment in the later image, earlier environment features located in the environment in the earlier image, and earlier object features located on the object in the earlier image" - Par 7, Lines 4-10
NOTE: Baig discloses capturing an earlier and later image of an environment using a camera. The environment includes object features located in an object in the environment. The camera used to capture the earlier and later images functionally corresponds to obtaining sensor data for a sequence of frames.
determining, based on the sensor data, a set of three-dimensional (3D) positions of a plurality of features for a first frame of the sequence of frames, the set of 3D positions including a 3D position of a feature corresponding to the object;
"When the first image is input to the system 1000, it starts with the ODPE module 1060 that uses training data 1050 for the object detection and provides the initial pose of the object and object map 1020, which is a set of 3D points belonging to feature points on the object." - Par 83, Lines 1-5
NOTE: Baig discloses obtaining an object map of an object in the first image. The object map contains the 3D points belonging to feature points of the object. Baig further discloses: “In other words, images in the image stream are analyzed directly for object movement by looking for object features 408, 408′, and their relative position.”, see 105, Lines 13-15. This shows obtaining the positions of the 3D points of the object in the environment.
determining that the object or content associated with the object is in motion based on a change in the feature between the first frame and a second frame;
"The processor is further configured to determine that the object has moved between the earlier image and the later image if the number of matched object features does not exceed a threshold." - Par 22, Lines 22-25
NOTE: Baig determines an object is in motion if the position of an object’s feature points in an earlier frame does not align with the estimated position in the later frame. Baig further discloses: "FIG. 4B shows an embodiment where object 404 has moved between the earlier image 400 and later image 400′. In contrast, FIG. 4C shows an embodiment where object 404 has not moved. As a result, there are no matched object features 438 in FIG. 4B, because the estimated object feature 428 is in a different location than the later object feature 408′." - Par 96, Lines 1-7.
"However, tracking might fail if a significant part of the environment starts moving. Therefore, moving objects in the environment are generally considered outliers and visual SLAM algorithms do not track them" - Par 80, Lines 6-12
NOTE: Baig discloses using SLAM-based camera pose tracking in one embodiment which considers feature points in motion as outliers and excludes them from SLAM’s mapping algorithm. Baig further discloses: “a coordinate system can be developed such that camera pose and object pose is tracked with respect to the world frame of reference (or a global coordinate system).”, see Par 77, Lines 15-17. This corresponds to tracking an image-based pose of the device in a 3D coordinate system. However, Baig does not disclose excluding the feature points in response to determining that an object is in motion.
Baig does not fully teach in response to determining that the object or content associated with the object is in motion, tracking an image-based pose of the device in a 3D coordinate system based, at least in part, on a subset of the plurality of features, the subset excluding one or more features associated with the object determined to be in motion. However, Mei teaches and in response to determining that the object or content associated with the object is in motion,
"if a dynamic target does exist, the SLAM system reshapes a data matrix to retain the descriptors of static feature points." - Par 6 , Line 7
NOTE: Mei discloses a SLAM system for determining dynamic objects (objects in motion) in a physical environment scene.
tracking an image-based pose of the device in a 3D coordinate system based, at least in part, on a subset of the plurality of features, the subset excluding one or more features associated with the object determined to be in motion.
"Step 1: acquiring the RGB image of the current frame of the camera for target detection, classifying all detected targets into static and potentially dynamic, calculating optical flow geometric constraints within the detection boxes of potentially dynamic targets, and determining the truly moving dynamic targets based on the constraint results...Step 5: Extract ORB feature points for the current frame image, and remove all feature points within the detection bounding boxes of truly moving dynamic targets; Step 6: Solve the camera pose of the current frame based on the remaining feature points and perform tracking;" - Par 5
NOTE: Mei discloses determining static and dynamic objects in a scene, determining whether the dynamic objects are truly moving, and if they are, remove all the feature points associated with the moving object to then determine the camera pose of the current frame. After the combination, the concept of removing the feature points associated with object in response to determining that the object is in motion to track the camera pose as taught by Mei can modify Baig’s system for using environment features to determine movement of an object in the environment. Baig’s system would be modified to determine an object that is in motion when observing that the predicted position of an object does not align in the next frame, exclude all of the features associated with the moving object, and track the camera pose of the HMD device with the remaining features.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Baig by incorporating the teachings of Mei to track an image-based pose of the device in a 3D coordinate system based on a subset of features wherein the subset excludes the features associated with the moving object. One would be motivated to make this combination to reduce the inconsistencies that occur when using moving objects in the environment as consideration in the camera localization process. Removing the feature points associated with a moving object would lead to the predicted result of a reliable camera pose estimation and mapping that best reflects the environment.
Regarding claim 1, the claim recites similar limitations to claim 14. Therefore, method claim 1 corresponds to the device disclosed in claim 14 and is rejected for the same reasons as obviousness as used above.
Regarding claim 20, the claim recites similar limitations to claim 14. Therefore, non-transitory computer-readable storage medium claim 20 corresponds to the device disclosed in claim 14 and is rejected for the same reasons as obviousness as used above.
Regarding claim 7, Baig in view of Mei teaches the method of claim 1. Baig as modified further teaches wherein determining that the object or content associated with the object is in motion is based on a pose of the device.
“After the camera movement is determined based on the change in location of environment features 416, the next step is to estimate object features 408′ in the later image 400′ using the earlier object features 408 and the determined camera movement. This determined camera movement is used to generate an estimation of movement of object feature 408, shown by arrow 418 in FIG. 4B.” – Baig Par 94 Lines 7-13
NOTE: Baig discloses that an estimation of where the object features in the earlier image frame should be located in the later image frame. When the object features points of the early and late image frames do not align, it is determined that the object is in motion. The estimation process considers both the object feature positions located in the earlier image frame and the camera movement/change in viewpoint, which would naturally include the camera pose, of the device.
Regarding claim 8, Baig in view of Mei teaches the method of claim 1. Baig as modified further teaches determining a change in a position of a viewpoint of the device during the sequence of frames;
“As can be seen when comparing FIG. 4A to FIGS. 4B and 4C, the camera has moved from the earlier image 400 to the later image 400′, resulting in a change of viewpoint (camera pose). The motion of the camera can be derived from the change in location (shown by arrows 416) from the earlier environment features 406 to the later environment features 406′.” – Baig, Par 93, Lines 7-11
and adjusting the image-based pose of the device in the 3D coordinate system based on the determined change in the position of the viewpoint.
“SLAM works by acquiring tracked features in the environment, and then as time progresses in the image stream, measuring an amount of movement of the feature points. This amount of movement can be used to calculate the motion of the camera, and the camera pose is updated with respect to the world frame of reference” – Baig, Par 78, Lines 8-13
NOTE: Baig further discloses: “a coordinate system can be developed such that camera pose and object pose is tracked with respect to the world frame of reference (or a global coordinate system).”, see Par 77, Lines 15-17. This shows that Baig’s system can determine camera movement/viewpoint change between an earlier and later image frame in which the camera pose of the device is updated/adjusted in the established coordinate system. The coordinate system would naturally be a 3D coordinate system since the camera’s pose would be based off its position in the 3D physical environment.
Regarding claim 12, Baig in view of Mei teaches the method of claim 1. Baig as modified further teaches wherein the sensor data comprises image data,
"The method includes acquiring an earlier image and a later image of the environment from an image stream captured by the camera. The method further includes identifying later environment features located in the environment in the later image, earlier environment features located in the environment in the earlier image, and earlier object features located on the object in the earlier image" - Par 7, Lines 4-10
NOTE: Baig discloses using a camera to capture the image frame of an earlier and later image of the environment. The camera can be understood as an image sensor that then provides the image data.
Regarding claim 13, Baig in view of Mei teaches the method of claim 1. Baig as modified further teaches wherein the device comprises a head-mounted device (HMD).
Embodiments of the present disclosure further include a head mounted display that determines and displays movement of an object in an environment. The head mounted display includes a camera that captures an image stream of the environment, including an earlier image (an image at a first time) and a later image (an image at a second time later than the first time).” – Par 22, Lines 1-7
Regarding claim 19, Baig as modified teaches the device of claim 14. Baig further teaches wherein the object is in motion for at least a portion of frames of the sequence of frames.
"FIG. 4B shows an embodiment where object 404 has moved between the earlier image 400 and later image 400′. In contrast, FIG. 4C shows an embodiment where object 404 has not moved. As a result, there are no matched object features 438 in FIG. 4B, because the estimated object feature 428 is in a different location than the later object feature 408′." – Baig Par 96, Lines 1-7.
Regarding claim 6, the claim recites similar limitations to claim 19. Therefore, method claim 6 corresponds to the device disclosed in claim 19 and is rejected for the same reasons as obviousness as used above.
Claim(s) 2 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baig, Mei, and Sagong (US 20220277476 A1), hereinafter Sagong.
Regarding claim 15, Baig in view of Mei teaches the device of claim 14. Baig as modified further teaches wherein determining that the object or content associated with the object is in motion based on a change in the feature between the first frame and the second frame
"The processor is further configured to determine that the object has moved between the earlier image and the later image if the number of matched object features does not exceed a threshold." - Par 22, Lines 22-25
comprises: determining a projected two-dimensional (2D) position of the feature on the second frame based on the 3D position of the feature determined for the first frame and the change in the pose of the device;
“From the location change of the environment features 416, the camera movement is derived. In other words, the camera movement will have a predictable mathematical relationship, such as a combination of 3D-3D rigid body transformation and 3D-2D projection, with the location change 416, and thus can be calculated from location change 416. After the camera movement is determined based on the change in location of environment features 416, the next step is to estimate object features 408′ in the later image 400′ using the earlier object features 408 and the determined camera movement. This determined camera movement is used to generate an estimation of movement of object feature 408, shown by arrow 418 in FIG. 4B. Based on this estimated movement 418, the estimated location of the object feature 428 is calculated. The estimated object feature 428 is where the processor expects the object feature 408 to have moved based on the camera movement estimation 418 alone.” – Par 94
NOTE: Baig further discloses determining camera movement (moving from a first frame to a second frame) and estimating the 2D position of an object’s feature as it should appear on the later image frame which would naturally require the obtained 3D position of the object features in the earlier frame.
and determining whether the object or content associated with the object is in motion based on the projected 2D position of the feature and an actual 2D position of the feature in the second frame.
"FIG. 4B shows an embodiment where object 404 has moved between the earlier image 400 and later image 400′. In contrast, FIG. 4C shows an embodiment where object 404 has not moved. As a result, there are no matched object features 438 in FIG. 4B, because the estimated object feature 428 is in a different location than the later object feature 408′." - Par 96, Lines 1-7
NOTE: Baig discloses that the determination of an object in motion is based on whether the position of the object features aligns with the estimated position of the object features in the later frame.
Baig as modified still does not teach wherein determining that the object or content associated with the object is in motion based on a change in the feature between the first frame and the second frame comprises: determining a change in a motion sensor-based pose of the device between the first frame and a second frame of the sequence of frames. However, Sagong teaches wherein determining that the object or content associated with the object is in motion based on a change in the feature between the first frame and the second frame comprises: determining a change in a motion sensor-based pose of the device between the first frame and a second frame of the sequence of frames
“The pose determination apparatus may determine the pose of the AR providing apparatus 110 based on image data obtained by an image sensor and motion data obtained by a motion sensor. The pose determination apparatus may use the image data and the motion data together, for example, to accurately and effectively estimate the pose of the AR providing apparatus 110.” – Par 56, Lines 1-6
NOTE: Sagong discloses a determining the pose of an AR providing apparatus 110 which is described to correspond to the pose of the user/camera, see Par 55, Lines 26-69. The pose of the user/camera in an image frame is determined by a motion data from a sensor. After the combination, the concept of using a motion data from a motion sensor to determine the pose of the device in an image frame as taught by Sagong can modify Baig system for obtaining an earlier and later image frame depicting the environment. This modification allows Baig’s system to determine the pose of the camera in each frame and determine any changes in the pose based on whether the motion data generated by the motion sensor is different in each frame.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Baig by incorporating the teachings of Sagong to determine a change in the motion sensor-based pose of the device between a first and second frame. One would be motivated to make this combination since Baig observes an earlier and later image frame of the environment to calculate the estimated positions of where an object should show up in a later frame. Using the pose obtained by the motion sensor of Sagong would assist in calculating the estimated position of the object in the later image frame. This would then lead to a predicted result of an accurate estimation of where the object feature points should be located in the image to clearly determine whether an object is in motion or not.
Regarding claim 2, the claim recites similar limitations to claim 15. Therefore, method claim 2 corresponds to the device disclosed in claim 15 and is rejected for the same reasons as obviousness as used above.
Claim(s) 3 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baig, Mei, and Yang et al (“Dynamic RGB-D visual odometry”), hereinafter Yang.
Regarding claim 16, Baig in view of Mei teaches the device of claim 14. Baig as modified does not teach wherein determining that the object or content associated with the object is in motion based on a change in the feature between the first frame and the second frame comprises: determining that a distance between a first 3D position of the feature for the first frame and a second 3D position of the feature for second frame exceeds a threshold. However, Yang teaches wherein determining that the object or content associated with the object is in motion based on a change in the feature between the first frame and the second frame comprises: determining that a distance between a first 3D position of the feature for the first frame and a second 3D position of the feature for second frame exceeds a threshold.
“The depth di of each feature is found by the corresponding RGB-D depth image. Then we get the 3D positions rwpi of the uncertain features of keyframe r in the world coordinate w… At the subsequent frames k, the uncertain features in keyframes are re-projected to the current frame, which forming feature pairs. Like in the keyframes, in the current frame we can also obtain the 3D positions… Ideally, if the motion estimation is sufficient accuracy and the scene is static, rwpi and kwpi are equal. If there are moving objects, the two may be different. So, we calculate the Euclidean distance between them…We use the threshold dis_threshold to determine whether they are similar” - Section 3, C. Mapping, Par 2-5
NOTE: Yang discloses obtaining a feature in a first frame captured by an RGB-depth camera. The 3D position of that feature is obtained using the depth information. The corresponding 3D position of the feature is also obtained in the next frame. The distance between the two 3D positions is obtained and a predetermined distance threshold determines if the positions are similar. If the threshold is exceeded, the object associated with the feature is in motion. After the combination, the concept of obtaining the 3D positions of the object feature between two frames and calculating the distance to see if it exceeds a threshold as taught by Yang can modify Baig’s device for determining movement of an object between an earlier and later image frame. Baig’s device observes the position of the object feature points in the earlier image to see if it aligns with the predicted position in the later image. This process is then modified to use the distance between the 3D position of the object feature in the earlier image and the estimated 3D position of the object feature in the later image to determine if the two positions are similar/aligned based on the distance threshold. This can then allow Biag’s device to determine whether the object is in motion.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Baig by incorporating the teachings of Yang to determine a distance between a first 3D position of the feature for the first frame and a second 3D position of the feature for second frame exceeds a threshold as part of the process to determine whether or not the object is in motion. One would be motivated to make this combination since a comparison of the distance between 3D positions of the object features in a first and second frame is a quantitative implementation that better provides an accurate indication of whether the object features point align with the estimated position. This would provide the predicted result of an improved discrimination between stationary and moving features.
Regarding claim 3, the claim recites similar limitations to claim 16. Therefore, method claim 3 corresponds to the device disclosed in claim 16 and is rejected for the same reasons as obviousness as used above.
Claim(s) 4-5 and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baig, Mei, and Tsurumi et al (WO 2023090213 A1), hereinafter Tsurumi.
Regarding claim 17, Baig in view of Mei teaches the device of claim 14, Baig as modified does not teach wherein the non-transitory computer-readable storage medium further comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: determining that a set of 3D positions of features corresponding to the object correspond to or approximately correspond to a planar structure. However, Tsurumi teaches wherein the non-transitory computer-readable storage medium further comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: determining that a set of 3D positions of features corresponding to the object correspond to or approximately correspond to a planar structure.
wherein the control unit calculates a display plane in three-dimensional space that includes the display area from at least three feature points included in the display area of the image” – Par 264, Lines 7-8
NOTE: Tsurumi teaches the camera of a headset display device showing an environment consisting of a displaying comprising TV images or game images. Tsurumi further discloses using feature points in the display area to calculate the display plane in 3D space. Planar structures can be defined as a display device such as a television as disclosed in Applicant’s specifications, see par 43, Lines 13-14). After the combination, the concept of using feature points corresponding to an object (such as moving content from a display) that corresponds to a planar structure (such as the display device) as taught by Tsurumi can modify Baig’s device for identifying 3D features points of objects in an environment to determine if the object is moving. The combination would then allow Baig’s device to identify the 3D positions of the features of an object corresponding to the planar device such as a tv or monitor.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Baig by incorporating the teachings of Tsurumi to determine that a set of 3D positions of features corresponding to the object correspond to or approximately correspond to a planar structure. One would be motivated to make this combination to clearly identify moving objects on a display screen which may create inconsistencies when performing SLAM pose mapping processes. By accurately determining the objects in motion, the system can easily identify what features to remove from the mapping process to create a more stable reflection of the environment.
Regarding claim 4, the claim recites similar limitations to claim 17. Therefore, method claim 4 corresponds to the device disclosed in claim 17 and is rejected for the same reasons as obviousness as used above.
Regarding claim 18, Baig in view of Mei and Tsurumi teaches the device of claim 17. Baig as modified further teaches wherein the non-transitory computer-readable storage medium further comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: determining 2D locations of the features corresponding to the planar structure;
“For example, if user U is facing the display device 200 (see Figure 1), the captured image M1 acquired by the terminal device 300 at time t1 includes the display device 200, as shown in the upper diagram of Figure 2. In this case, the information processing device 100 acquires a plurality of feature points P11 outside the area including the display screen of the display device 200 (hereinafter also referred to as the display area). Furthermore, the information processing device 100 acquires multiple feature points P12 within the display area.” – Tsurumi Par 36
NOTE: Tsurumi discloses a process of acquiring multiple feature points of both objects outside and inside of a planar structure such as a display device. Tsurumi further discloses a process of calculating a 2D mask region over a captured image and selecting the feature points belonging outside of the mask region and deleting the features that are within the mask region which correspond to identifying the position of the feature points, see Tsurumi Par 114-116
and filtering a subsequent sequence of frames based on the 2D locations of the features corresponding to the planar structure.
“In this way, the information processing device 100 estimates the position of the terminal device 300 using information from the display area on which the display device 200 can display moving images. As a result, even if the display device 200 is displaying a moving image, the information processing device 100 can avoid using the feature points corresponding to the display device 200 for self-localization estimation” – Par 48
NOTE: After the combination, the concept of avoiding using the feature points (filtering out) corresponding to the display device (planar structure) for self-localization estimation as taught by Tsurumi can modify Baig’s system for identifying objection in motion between captured image frames. This modification takes Baig’s process for excluding feature points of objects in motion to then specifically filter out/exclude the object feature points corresponding to a planar structure such as a display device displaying content in motion.
Regarding claim 5, the claim recites similar limitations to claim 18. Therefore, method claim 5 corresponds to the device disclosed in claim 18 and is rejected for the same reasons as obviousness as used above.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baig, Mei, Uberti et al (US 20190392642 A1), hereinafter Uberti.
Regarding claim 9, Baig in view of Mei teaches the method of claim 1. Baig as modified further teaches presenting a view of an extended reality (XR) environment on a display,
“The HMD 100 includes a wearing belt 90 wearable on the head of the user, a display section 20 that displays an image, and a control section 10 that controls the display section 20. The display section 20 causes the user to sense a virtual image in a state in which the display section 20 is worn on the head of the user. The display section 20 causing the user to sense the virtual image is referred to as “display AR” as well. The virtual image sensed by the user is referred to as AR image as well.” – Par 44
NOTE: Baig discloses the HMD device displaying a virtual image/augmented reality image of an environment. Extended reality (XR) is a well-known term that encompasses augmented reality AR, virtual reality VR, and mixed reality MR.
wherein the view of the XR environment comprises
"The method includes acquiring an earlier image and a later image of the environment from an image stream captured by the camera. The method further includes identifying later environment features located in the environment in the later image, earlier environment features located in the environment in the earlier image, and earlier object features located on the object in the earlier image" - Par 7, Lines 4-10
NOTE: Baig’s device is configured to capture image frames of the physical environment which consists of an object.
Baig as modified does not explicitly teach that wherein the view of the XR environment comprises virtual content and at least a portion of the physical environment, wherein the portion of the physical environment includes the object. However, X teaches wherein the view of the XR environment comprises virtual content and at least a portion of the physical environment, wherein the portion of the physical environment includes the object
“FIG. 5 shows an example of a similar scene to that in FIG. 4, in which a virtual object 502, in this case, a tyrannosaurus rex (T-Rex), is shown as being present in a user's physical, real-world environment.” – Par 47
NOTE: Uberti discloses receiving input of a virtual object and displaying it to the user’s real-world environment. Uberti displays this view of the environment with the virtual object through a smartphone, but this augmented reality display could also be implemented using the HMD as described in par 21. Fig. 5 shows the augmented reality image of a tyrannosaurus rex overlayed on the physical environment. After the combination, Baig’s device would be modified to include virtual content into the physical environment.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Baig by incorporating the teachings of Uberti to have the XR environment comprise virtual content In the physical environment. This would create an augmented reality experience for the user in the physical environment. Baig’s device already observes a physical environment using an HMD and looks specifically for moving objects in the real-world environment to exclude the feature points that may affect the tracking/mapping process as the user moves within the environment. This combination would allow Baig’s system to apply its process to create a more stable presentation of a virtual object within the physical environment.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baig, Mei, Uberti and Hare et al (US 20190178654 A1), hereinafter Hare.
Regarding claim 10, Baig in view of Mei and Uberti teaches the method of claim 9. Baig as modified does not teach wherein the virtual content is adjusted based on determining to exclude the one or more features associated with the object determined to be in motion. However, Hare teaches wherein the virtual content is adjusted based on determining to exclude the one or more features associated with the object determined to be in motion.
“and determining second coordinates of the virtual object based on the second estimated coordinates of the feature and the displacement of the virtual object relative to the feature.” – Par 25
NOTE: Hare discloses that the displaying of the virtual object is changed/displaced from its first position to a second potion after a feature-coordinate re-estimation process. The displacement can be understood as an adjustment to the virtual object. The re-estimation process considers only the feature points of stationary objects and excludes the features associated with objects in motion: “For example, relevant data (e.g., stationary features) to determine position and orientation may be identified and separated from non-relevant data (e.g., moving object in frame). The optimization algorithms may then be applied to the relevant data only or separately.”, see Par 181, Lines 6-10.. After the combination, the concept of adjusting the position of a virtual object based on excluding features associated with an object determined to be in motion as taught by Hare can be applied to the virtual object added to the physical environment as taught by Uberti. This modification is further added to Baig to enable the device to adjust the position of the virtual object between frames when excluding the frames associated with the object in motion.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Baig by incorporating the teachings of Hare to adjust the virtual content based on determining to exclude the one or more features associated with the object determined to be in motion. One would be motivated to make this combination to improve stability of the virtual content/object that may be affected by the inconsistencies in the localization process from moving objects in the environment. By excluding the features associated with the moving objects during the localization process, the virtual content can properly be displayed as the user changes their viewpoint of the physical environment.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baig, Mei, and Bisain et al (US 11727576 B2) hereinafter Bisain.
Regarding claim 11, Baig in view of Mei teaches the method of claim 1. Baig as modified does not teach wherein the sensor data is determined from an image sensor signal based on a machine learning model configured to identify image portions corresponding to the object. However, Bisain teaches wherein the sensor data is determined from an image sensor signal based on a machine learning model configured to identify image portions corresponding to the object
“wherein identifying the portion of the image that includes the depiction of the dynamic object includes: identifying, using at least a first trained neural network, that the image includes the depiction of the dynamic object, and identifying, using at least a second trained neural network in response to identification that the image includes the depiction of the dynamic object, the portion of the image that includes the depiction of the dynamic object.” – Claim 27
NOTE: Bisian discloses a device receiving an image of an environment from an image sensor wherein the image depicts static and dynamic objects in the environment. A trained neural network is used as part of the identification of dynamic objects (objects in motion) in the image, see Abstract. This functionally corresponds to sensor data being determined from an image sensor signal based on a machine learning model configured to identify image portions corresponding to the object.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Baig by incorporating the teachings of Bisain to have the sensor data be determined from an image sensor signal based on a machine learning model configured to identify image portions corresponding to the object. One would be motivated to make this combination to improve Baig’s existing process for identifying objects in motion by using a machine learning algorithm that learns how to find objects in motion in an image. The machine learning algorithm provides the advantage of a complex model that can learn and adapt to images that may be difficult to determine if an object is in motion in the image.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID V. NGUYEN whose telephone number is (571)272-6111. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID VAN NGUYEN/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights