Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objection
Claim 9 objected to because of the following informalities: misspelling. Claim 9 cites “audio queues” rather than “audio cues” as cited in the specification, paragraph 46. Appropriate correction is required.
Claim Rejection – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 5, 6, 11, 12, 13, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shroff (US20230032467A1) hereinafter referenced as Shroff, in view of the following: Peruch (US9912862B2), hereinafter referenced as Peruch and Hofer (US2004189849A1) hereinafter referenced as Hofer.
Regarding claim 1, Shroff teaches the following:
A method for obtaining information regarding a scene using a display free body wearable computing device, the method comprising: identifying a scope of interest in the scene as indicated by a user of the display free wearable computing a device;
“when executed by the one or more processors, cause the system to perform a method. The method includes receiving from a user of a smart glass [17] one or more wearable devices 100 - 1 (e.g., a smart glass) [26] … read user gesture to identify a region of interest in a scene viewed through a smart glass as disclosed above [21] … The system may or may not have a display for the user [22].” (Shroff, [17, 21, 22, 26])
Shroff teaches a method for obtaining information regarding a scene using wearable devices, the method comprising: identifying a region of interest in the scene as indicated by a user of a device.
Shroff fail to teach the following: obtaining an image capture plan based images; and actively guiding movement of the user based on the image capture plan to obtain a set of images depicting portions of the scene.
Peruch does and teaches the following:
obtaining an image capture plan based images based on the scope of interest;
“compute a guidance map in accordance with the one or more initial images to identify one or more next poses” (Peruch, [07])
Peruch teaches computing a guidance map based the one or more initial images identify next poses. The guidance map is inherently derived from initial images which necessarily reflect the portion of the scene being capture (reads on scope of interest).
obtaining, while actively guiding the movement of the user based on the image capture plan and using at least two image sensors of the system, stereo images depicting portions of the scene, each of the stereo images comprises a pair of images of the portions of the scene, the pair of images being captured simultaneously using the at least two image sensors; and
“FIG. 2A is a block diagram of a scanning system according to one embodiment of the present invention. [6] … In some embodiments, the scanning system 10 includes a camera, a display screen, memory, and a processing unit or processor. The user 14 can freely move the scanning system 10 over a path that captures images of the scene from multiple different poses, thereby resulting in multiple views. The scanning system 10 can assist the user in moving the scanning system over a path that efficiently provides good results. [28] … “FIG. 2A is a block diagram of a scanning system as a stereo depth camera system according to one embodiment of the present invention.” [38] … The first camera 102 and the second camera 104 include corresponding image sensors 102a and 104a [39] … the processor may control cameras 102 and 104 to capture images at substantially the same time and use the captured images to compute a depth image using image disparity techniques as described above [60] … As discussed above, an incremental coverage map) ΔΩ.sup.N+1(p) is a function that associates each possible camera pose p from which a new view could be obtained, with the incremental coverage that would be provided by this new view … The incremental coverage map may be used to provide guidance as to where to move the scanning system 10 in order to increase global coverage. [76]” (Peruch, [6, 28, 76])
PNG
media_image1.png
657
882
media_image1.png
Greyscale
Peruch teaches obtaining while actively guiding the movement of the user using a scanning system to assist the user in moving the scanning system over an incremental coverage map that efficiently provides good results (reads on obtaining, while actively guiding the movement of the user based on the image capture plan). The scanning system includes two image sensors, see FIG 2A which is a block diagram of a scanning system. (reads on using at least two image sensors of the system). The scanning system captures images of the scene from multiple different poses, thereby resulting in multiple views (reads on a set of stereo images depicting portions of the scene, each of the images comprises a pair of images of the portions of the scene) wherein the processor may control cameras and to capture images at substantially the same time (reads on the pair of images being captured simultaneously using the at least two image sensors. The images are taken at the same time by two different cameras thus resulting in capturing images of the scene from multiple different poses and views. This is the same as a set of stereo images.
stereo images obtained by the at least two image sensors to obtain a set of images depicting the portions of the scene
“FIG. 2A is a block diagram of a scanning system according to one embodiment of the present invention. [6] … The user 14 can freely move the scanning system 10 over a path that captures images of the scene from multiple different poses, thereby resulting in multiple views. The scanning system 10 can assist the user in moving the scanning system over a path that efficiently provides good results. [28] … The first camera 102 and the second camera 104 include corresponding image sensors 102a and 104a [39] … the processor may control cameras 102 and 104 to capture images at substantially the same time and use the captured images to compute a depth image using image disparity techniques as described above [60]” (Peruch, [6, 28, 39, 60])
PNG
media_image1.png
657
882
media_image1.png
Greyscale
The scanning system captures images of the scene from multiple different poses, thereby resulting in multiple views (reads on a set of stereo images depicting the portions of the scene) captured by image sensors corresponding to the first camera and the second camera (reads on images obtained by the at least two image sensors to obtain).
Peruch BASE is analogous art with respect to Shroff because they are from the same field of endeavor, namely image capturing and processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff to create a method/system for obtaining information regarding a scene using a wearable computing device, the method comprising: identifying a region of interest in the scene as indicated by a user of a device; computing a guidance map based the one or more initial images identify next poses, wherein this system guides users to capture optimal scan data by following a preset image capture plan and a live coverage map. Using a device with at least two image sensors to take simultaneous pairs of stereo images from various angles to capture a stereo images of the scene. A person of ordinary skill in the art would do such in order to improve image scanning efficiency.
Shroff in view of Peruch does not teach active guiding movement of the user based on the image capture plan to obtain a set of images depicting portions of the scene; and stitching each of the images obtained depicting the portions of the scene
But Hofer does. Hofer teaches the following:
actively guiding movement of the user based on the image capture plan to obtain a set of images depicting portions of the scene.
“Disclosed is a panoramic sequence guide that can be used to guide a user in composing and capturing images to be used to create a composite panoramic image. In use, the guide tracks a viewed scene, determines the proper position of a composition guide relative to the viewed scene” (Hofer, [06])
Hofer teaches actively guiding user movement via a dynamic positioned composite guide, based on a planned sequence of scene portions to be captured, to obtain set images of the scene.
stitching each of the images obtained depicting the portions of the scene
“If the user captures the second image when the camera is so aligned, there will presumably be adequate overlap between the two images (e.g., approximately 1/3 of the image width) to later join them together using an appropriate "stitching" program.” (Hofer, [3])
Hofer teaches stitching the overlap between the two images of a scene captured by user.
Hofer BASE is analogous art with respect to Shroff in view of Peruch because they are from the same field of endeavor, namely image capturing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch to create a method for obtaining information regarding a scene using smart glasses, the method comprising: identifying a region of interest in the scene as indicated by a user of a device; computing a guidance map based the one or more initial images identify next poses; actively guiding user movement via a dynamic positioned composite guide, based on a planned sequence of scene portions to be captured, to obtain set images of the scene, and to stitch the overlap between the two images of a scene captured by user. A person of ordinary skill in the art would do such in order to improve engagement, usability, and rewarding experiences.
Regarding claim 2, Shroff in view of Peruch and Hofer teach the method of claim 1 and additionally teaches the following. Shroff teaches:
wherein obtaining, using at least one camera of the display free body wearable computing device, a first image, and depicting: a first portion of the scene, a gesture introduced into the scene by the user;
“the camera captures the scene, including some example hand gestures from the user. The frames including hand gestures” (Shroff, [25])
Shroff teaches capturing, using at least one camera of the wearable computing device, a first image depicting: the frame of the scene, and a gesture introduced into the scene by the user
and identifying, based on the gesture and the first portion of the scene, the scope of the interest in the scene.
“identify the region of interest based on a reading of the hand gesture from the user.” (Shroff, [25])
Shroff teaches identifying, based on the gesture and the frame of the scene, the region of the interest in the scene.
Peruch continues and teaches:
capturing the first image being separate from the stereo images and depicting: a first portion of the scene
“Referring to FIG. 3, in operation 302, the processor (e.g., the host processor 108 in FIG. 2A or the processor 210 working in concert with the controller 108 of FIG. 2B) controls the scanning device to capture an initial image. [60] … the initial image may be captured along an optical axis perpendicular to the ground plane that the object is resting on (e.g., a top-view of the object). [61] … Because this quick scan is used to initialize the process, issues of low coverage and under-sampling associated with this initial scan will be addressed and corrected through the process. [62]” (Peruch, [60-62])
Peruch teaches of capturing the initial image being separate from the stereo images along an optical axis perpendicular to the ground plane that the object is resting on (reads on depicting: a first portion of the scene). This is a part of initialization the process which is separate from the rest of the scanning process for the stereo images (reads on capturing the first image being separate from the stereo images).
stereo images being captured by the at least one camera being one of the at least two image sensors;
“The first camera 102 and the second camera 104 include corresponding image sensors 102a and 104a [39] … the processor may control cameras 102 and 104 to capture images at substantially the same time and use the captured images to compute a depth image using image disparity techniques as described above [60]” (Peruch, [39, 60])
The scanning system captures images of the scene from multiple different poses, thereby resulting in multiple views wherein the processor controls cameras to capture images at substantially the same time. The images are taken at the same time by two different cameras thus resulting in capturing images of the scene from multiple different poses and views. This is the same as a set of stereo images.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff to using a device with at least two image sensors, take simultaneous pairs of stereo images from various angles to capture a stereo image of the scene, and additionally capturing the initial image along an optical axis perpendicular to the ground plane that the object is resting on, wherein this is a part of initialization the process. A person of ordinary skill in the art would do such in order to improve image scanning efficiency.
Claim 19 is rejected using the same rationale or bases as applied to claim 2.
Regarding claim 5, Shroff in view of Peruch and Hofer teaches the method of claim 2, and additionally teaches the following. Peruch teaches:
wherein obtaining the image capture plan comprises: identifying, based on the scope of the interest, a number of images usable to generate a three-dimensional model having a desired level of quality;
“compute a guidance map in accordance with the one or more initial images to identify one or more next poses; … and output the images captured by the camera to generate a three-dimensional model [07] ... To build a successful (or sufficiently useful) 3D model, aspects of embodiments of the present invention are directed to providing assistance or guidance to suggest a good path to move the camera during scanning [46]” (Peruch, [07, 46])
Peruch teaches wherein computing the guidance map comprises: identifying, based on the one or more initial images of the object to be scanned (reads upon the scope of the interest) a number of poses usable to generate sufficient three-dimensional model;
for each image of the number of images, identifying a location and field of view for obtaining the respective image based on the three-dimensional model having the desired level of quality.
“the system determines whether scanning should continue or whether there is sufficient data to generate a good 3D model. As discussed above, an incremental coverage map) ΔΩN+1 (p) is a function that associates each possible camera pose p from which a new view could be obtained … Note that a pose p=(x, O) represents both the location x and orientation O of the camera system at the time of capture.” (Peruch, [95])
Peruch teaches for each pose of the number of poses wherein each pose is defines the camera is placed and oriented at the moment of an image is captured (reads on for each image of the number of images), identifying a location and orientation for obtaining the respective pose based having sufficient data to generate a good 3D model.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer with the feature of Peruch to wherein computing the guidance map comprises: identifying, based on the one or more initial images of the object to be scanned, which reads upon the scope of the interest, a number of poses usable to generate sufficient three-dimensional model; and for each pose of the number of poses, identifying a location and orientation for obtaining the respective pose based having sufficient data to generate a good 3D model. A person of ordinary skill in the art would do such in order to improve computationally and power efficient for image processing.
Regarding claim 6, Shroff in view of Peruch and Hofer teaches the method of claim 5, and additionally teaches the following. Peruch teaches:
wherein obtaining the image capture plan further comprises: obtaining a motion plan based on the number of images and the location and field of view of each image of the number of images, wherein the motion plan when performed by the user, enables the set of images that allow for the three-dimensional model having the desired level of quality to be generated to be captured using the at least one camera.
“compute a guidance map in accordance with the one or more initial images to identify one or more next poses [07] … The user 14 can freely move the scanning system 10 over a path that captures images of the scene from multiple different poses, thereby resulting in multiple views. The scanning system 10 can assist the user in moving the scanning system over a path that efficiently provides good results [49] … the processor may control cameras 102 and 104 to capture images at substantially the same time and use the captured images to compute a depth image using image disparity techniques as described above [60] … the system determines whether scanning should continue or whether there is sufficient data to generate a good 3D model. As discussed above, an incremental coverage map) ΔΩN+1 (p) is a function that associates each possible camera pose p from which a new view could be obtained, … Note that a pose p=(x, O) represents both the location x and orientation O [95]” (Peruch, [ 07, 49, 60, 95])
The system computes the next poses using prior images and a coverage map, which defines a location and orientation of the pose. These poses collectively determine where and how the images should be captured. The system guides the user along a path through these poses. Peruch teaches generating a sequence of camera pose (reads on image capture plan) based on analysis of the previously captured images via a coverage map (reads on the number of images), where each pose specifies a location and orientation (reads and the location and field of view of each image of the number of images) providing guidance to the user to follow a path through the poses. As the user moves along this path, images are captured from multiple viewpoints forming a set of suitable images for the 3D model. The scanning system utilizes two cameras to capture the images.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer with the feature of Peruch wherein computing the guidance map comprises: identifying, based on the one or more initial image, a location, and orientation for obtaining the respective pose, a plan, when performed by the user, enables a set of views to be captured using the at least one camera that allow for sufficient data to generate a good 3D model. A person of ordinary skill in the art would do such for more computationally and power efficient image processing.
Regarding claim 11, Shroff in view of Peruch and Hofer teaches the method of claim 1, and additionally teaches the following. Peruch teaches:
further comprising: providing computer-implemented services using the set of images
“The remote processor 18, typically having much higher processing power and storage capabilities than the scanning system 10, can quickly optimize and finalize the 3D scan, and can provide this service to multiple scanning systems.” (Peruch 51)
Peruch teaches providing a remote processor services using scans.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer with the feature of Peruch to further comprise: providing computer-implemented services using the set of images. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Regarding claim 12, Shroff in view of Peruch and Hofer teaches the method of claim 11, and additionally teaches the following. Peruch teaches:
wherein providing the computer-implemented services comprises: generating a three-dimensional model using the set of images;
“The multiple views are combined to generate a 3D model [50] … generating the 3D model may be computationally expensive…transmit raw or partially processed data… to a remote (or “offline”) processor 18 [51]” (Peruch, [50, 51])
Peruch teaches providing the remote processor comprises: generating a three-dimensional model using the set of views;
obtaining an insight using the three-dimensional model;
“The resulting 3D model can then be used to estimate the available space or volume 12a of the cargo area of the truck” (Peruch, [50])
Peruch teaches obtaining an insight regarding the available space using the resulting three-dimensional model;
and using the insight to convey enhanced information regarding the scene to the user.
“The results can be presented to the users directly from the servers …After capturing the 3D scan of the boxes, as shown in FIG. 7A, resulting 3D model can be analyzed to count the number of boxes that are present. (Peruch, [51])
Peruch teaches using the insight to convey from the resulting 3D enhanced information regarding the scene to the user moreover the analyzing the number of boxes within the scene.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer with the feature of Peruch for providing the remote processor comprises: generating a three-dimensional model using the set of views; obtaining an insight regarding the available space using the resulting three-dimensional model; using the insight to convey from the resulting 3D enhanced information regarding the scene to the user moreover the analyzing the number of boxes within the scene. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Regarding claim 13, Shroff in view of Peruch and Hofer teaches the method of claim 1 and additionally teaches the following. Shroff teaches:
wherein the display free body wearable computing device comprises: an integrated sensing and interaction component that comprises the at least two image cameras of the display free body wearable computing device and that is adapted to: be positioned symmetrically on two portions of a user’s head, be positioned between ears and eyes of the user, and capture the stereo images;
“wearable devices with video camera accessories…In some embodiments, a system includes a smart glass [22] … a smart glass 300 to help the user adjust an FOV 338 of a first forward-looking camera 315 - 1 based on a region of interest for image capture, according to some embodiments. The scene 310 may be captured also by a second forward-looking camera 315 - 2 (hereinafter, first and second forward-looking cameras 315 - 1 and 315 - 2 will be referred to, hereinafter, as ‘cameras 315’ [34]”) (Shroff [28, 34])
PNG
media_image2.png
477
692
media_image2.png
Greyscale
FIG 3. (Shroff)
Shroff teaches the wearable computing device comprises: smart glasses adapted to: be positioned symmetrically on two portions of a user’s head, be positioned between ears and eyes of the user, and capture region of interest based on the field of view of the user and scene;
an integrated computing, powering, and securing portion;
“Smart glass 100 - 1 may include a frame 105 … Processor 112 is configured to identify a region of interest (ROI) of user 101” (Shroff, [27])
PNG
media_image3.png
405
306
media_image3.png
Greyscale
FIG. 1. (Shroff)
Shroff teaches of an integrated processor secured on the frames which provides the computational power for the computer system of the smart glasses;
and an adjustment member adapted to position the integrated sensing and interaction component with respect to the integrated computing, powering, and securing portion.
“Smart glass 100 - 1 may include a frame 105. A camera 115 (e.g., forward-looking) is mounted on frame 105. A sensing device 128 … Processor 112” (Shroff, [27])
PNG
media_image4.png
405
306
media_image4.png
Greyscale
FIG. 1. (Shroff)
Shroff teaches a frame adapted to position the multiple sensors with respect to the integrated processor which provides the computational power for the computer system of the smart glasses.
Peruch also teaches:
at least two image sensors of the device
“The first camera 102 and the second camera 104 include corresponding image sensors 102a and 104a” (Peruch, [39])
PNG
media_image1.png
657
882
media_image1.png
Greyscale
Peruch teaches of two cameras which include corresponding image sensors.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer with the feature of Peruch to use at least two image sensors corresponding to the two cameras which take simultaneous pairs of stereo images from various angles. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable Shroff in view of Peruch, Hofer, and Kumar (US20160091975A1), hereinafter referenced as Kumar.
Regarding claim 3, Shroff in view of Peruch and Hofer teaches the method of claim 2 but fail to teach wherein, in an instance of the obtaining of the first image where the gesture indicates interest of the user in an object present in the scene: the scope of the interest is limited to the object. Kumar teaches the following:
wherein, in an instance of the obtaining of the first image where the gesture indicates interest of the user in an object present in the scene: the scope of the interest is limited to the object.
“a region of interest may be selected by enclosing an object with a user's hand or hands. [05] … Once the region of interest is properly identified, the object within the region of interest may be cropped” (Kumar, [05, 18])
Kumar teaches an instance wherein, in an instance of the obtaining of the first image where the gesture indicates a region of interest is localized to the object in the region of interest: the region of interest is cropped based on the object.
Kumar BASE is analogous art with respect to Shroff in view of Peruch and Hofer because they are from the same field of endeavor, namely image capturing and processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer wherein an instance of the obtaining of the first image where the gesture indicates a region of interest is localized to the object in the region of interest: the region of interest is cropped based on the object. A person of ordinary skill in the art would do to improve user experience.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shroff in view of Peruch, Hofer, and Xu (US20130076913A1), hereinafter referenced as Xu.
Regarding claim 4, Shroff in view of Peruch and Hofer teaches the method of claim 2, and additionally teaches the following. Shroff teaches:
wherein, in an instance of the obtaining of the first image where the gesture indicates general interest: the scope of the interest is the field of view.
“the camera captures the scene … and identify the region of interest based on a reading of the hand gesture from the user … the hand gesture may be to increase, expand, pan, or shift an FOV of one or more of cameras 115 according to an ROI of user 101” (Shroff, [25, 30])
Shroff teaches wherein, in an instance of the obtaining of the first image where the gesture indicates general interest: the scope of the interest is the field of view.
Shroff in view of Peruch and Hofer fail to teach the scope of the interest is all of the scene.
But Xu does. Xu teaches the following:
the scope of the interest is all of the scene
“spectral sensor 604 is focused on a pre-defined area of interest such as all of scene 601 in field of view 614A and, upon a triggering event having occurred” (Xu, [62])
Xu teaches the area of interest is all of scene.
Xu BASE is analogous art with respect to Shroff in view of Peruch and Hofer because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer wherein, in an instance of the obtaining of the first image where the gesture indicates general interest: the area of interest is all of scene. A person of ordinary skill in the art would do such in order to improve computationally and power efficient for image processing.
Claim(s) 7, 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shroff in view of Peruch and Hofer and Said (US2017307719A1), hereinafter referenced as Said.
Regarding claim 7, Shroff in view of Peruch and Hofer teaches the method of claim 1, and additionally teaches the following. Peruch teaches:
wherein actively guiding movement of the user comprises: monitoring the movement of the user;
“Aspects of embodiments of the invention are directed to monitoring the coverage and the measurement density of different patches of surfaces of the subject (e.g., an object or scene) being scanned.” (Peruch, [78])
Peruch teaches wherein monitoring of the user comprises: monitoring the coverage and the measurement density of different patches of surfaces of the subject (e.g., an object or scene) being scanned by the user.
an image capture plan
“compute a guidance map in accordance with the one or more initial images to identify one or more next poses” (Peruch, [07])
Peruch teaches a guidance map.
a motion plan
“The user 14 can freely move the scanning system 10 over a path that captures images of the scene from multiple different poses.” (Peruch, [49])
Peruch teaches a path plan.
Shroff in view of Peruch and Hofer do not teach the following: comparing the movement of the user to a motion plan indicated by the image capture plan; and in an instance of the comparing where the movement of the user diverges from the motion plan: providing, to the user, sensory feedback that indicates a change in the movement of the user, the change being adapted to conform the movement of the user to the motion plan.
Said does. Said teaches the following:
comparing the movement of the user to a motion plan indicated by the calculated route;
“The audio navigation system 25 may learn of a user deviation through the mobile device 100 encountering the beacon 42 that is not along the calculated route.” (Said, [66])
Said teaches comparing the mobile device, which is carried by the user as they walk, to beacons along calculated route;
in an instance of the comparing where the movement of the user diverges from a motion plan: providing, to the user, sensory feedback that indicates a change in the movement of the user, the change being adapted to conform the movement of the user to the motion plan.
“Audio may be generated by a text-to-speech component …When in a navigation mode, the system may give turn-by-turn directions while continuously updating the user on her present location and recalculating the route in the case the user deviates from the calculated route … At this point, the audio navigation system 25 may calculate a new route from the user's current location to the user to their desired destination.” (Said, [13-14, 66])
Said teaches of when the user deviates from the calculated route, the audio navigation system will provide turn-by-turn directions indicating the user deviated from the calculated route and the change is being recalculated based on user’s current location to the user to their desired destination.
Said teaches in an instance of the comparing where the user deviation from the calculated route, providing, to the user, audio navigation system feedback which includes turn-by-turn directions that indicates the user deviates from the calculated route, the change being recalculated to conform the user's current location to the user to their desired destination.
Said BASE is analogous art with respect to Shroff in view of Peruch and Hofer because they are from the same field of endeavor, namely image processing and guided movements. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer wherein monitoring of the user comprises: monitoring the coverage and the measurement density of different patches of surfaces of the subject (e.g., an object or scene) being scanned by the user; comparing the mobile device, which is carried by the user as they walk, and beacons along calculated route, guidance map guidance map; and in an instance of the comparing where of the user deviation from the calculated route; providing, to the user, an audio navigation system feedback which includes turn-by-turn directions that indicates the user deviates from the path plan, the change being recalculated to conform the user's current location to the user to the path plan to the desired destination. A person of ordinary skill in the art would do such in order to improve user experience and useability.
Regarding claim 9, Shroff in view of the following: Peruch, Hofer, and Said teach the method of claim 7 teaches the method of claim 7, and additionally teaches the following. Said teaches:
wherein the sensory feedback comprises audio queues that convey the change in the movement to the user.
“Audio may be generated by a text-to-speech component that converts text generated by the system into audio announcements that are played back to the user [13] … the user deviates from the calculated route, the mobile device application 141 will immediately alert the user and the audio navigation system 25 will calculate a new route from the user's current location.” (Said, [13, 78])
Said teaches the audio navigation system comprises text-to-speech audio alerts that convey the user deviated from the route.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of the following: Peruch, Hofer, and Said with the feature of Said which teaches wherein audio navigation system comprises alerts that convey the user deviated from the route. A person of ordinary skill in the art would do such in order to improve user experience and useability.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable Shroff in view of Peruch and Hofer, Said, in view of Sommer (US11709068B2), hereinafter referenced as Sommer.
Regarding claim 8, Shroff in view of the following: Peruch, Hofer, and Said teach the method of claim 7, but does not teach the following: wherein the sensory feedback comprises spatial audio adapted to focus attention of the user on the scope of interest. But Sommer does. Sommer teaches:
wherein the sensory feedback comprises spatial audio adapted to focus attention of the user on the scope of interest.
“the system may recalculate a new path to the target location and begin directing the user on the new path using the spatial audio navigation methods.” (Sommer, 87)
Sommer teaches wherein the system comprises spatial audio navigation methods adapted to focus on the scope of interest of the user on the new path.
Sommer BASE is analogous art with respect to Shroff in view of Peruch and Hofer, Said because they are from the same field of endeavor, namely 3D image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Sommer to wherein the system comprises spatial audio navigation methods adapted to focus on the scope of interest of the user on the new path. A person of ordinary skill in the art would do such in order to improve user experience and useability.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shroff in view of Peruch, Hofer, Said, Li (US20170308772A1), hereinafter referenced as Li, and Stathacopoulos (CN109997094B), hereinafter referenced as Stathacopoulos.
Regarding claim 10, Shroff in view of the following: Peruch, Hofer, and Said teaches the method of claim 7, and additionally teaches the following. Hofer teaches:
wherein actively guiding movement of the user comprises: as the set of the images are obtained: comparing each image of the set of images to each other to determine the direction of capture;
“During tracking, relatively low resolution images of the viewed scene are captured in rapid succession and compared to determine the "movement" of objects in the scene relative to the camera orientation” (Hofer, [29])
Hofer teaches wherein tracking: as the set of images are captured in rapid succession: compared each image of the set of images to each other to determine the direction of capture.
Shroff teaches:
providing, to the user, second sensory feedback that indicates a change in the movement of the user’s head, the change being adapted to enable adjusting the user’s head.
“a feedback to the user based on a relative position between the region of interest and the field of view of the first forward-looking camera, the feedback intended to have the user adjust a head position relative to the object of interest [17] ... the system could provide feedback to the user indicating that ROI 238 is outside the FOV of one of forward-looking cameras 215 …This indication could be provided, for example, using audio 207 [32]” (Shroff, [17, 32])
Shroff teaches providing, to the user, audio feedback that indicates a change in the movement of the user’s head regarding the region of interest being outside the field of view, the change being adapted to enable adjusting the user head relative to the object of interest.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of the following: Peruch, Hofer, and Said with the features of Shroff and Hofer wherein tracking: as the set of images are captured in rapid succession: compared each image of the set of images to each other to determine the direction of capture; and providing, to the user, audio feedback that indicates a change in the movement of the user’s head regarding the region of interest being outside the field of view, the change being adapted to enable adjusting the user head relative to the object of interest. A person of ordinary skill in the art would do such in order to improve user experience.
Shroff in view of the following: Peruch, Hofer, and Said fail to teach the following: comparing each image of the set of images to a quality standard; in an instance of the comparing where an image of the set of images does not meet the quality standard: discarding the image of the images; providing, to the user, notification that indicates a change in user location, the change being adapted to enable obtaining of a replacement image for the image of the images.
But Li does. Li teaches
comparing each image of the set of images to a quality standard;
“comparing the qualities of a plurality of images when the number of images in the any same group is plural, and ranking the plurality of images according to the comparison result [22] … the quality of an image can be judged according to the characteristics (for example, definition) of the image so as to classify the image based on the quality of the image... the analyzing unit comprises a calculating unit for calculating the definition degree of the any image according to a preset definition degree calculating function [31-32]” (Li, [22, 31-32])
Li teaches comparing each image to definition degree.
in an instance of the comparing where an image of the set of images does not meet the quality standard: discarding the image of the images; and
“the quality of an image can be judged according to the characteristics (for example, definition) of the image so as to classify the image based on the quality of the image, images of which the category meets quality requirements are recommended to be saved, and images of which the category does not meet the quality requirements are recommended to be deleted.” (Li, [32])
Li teaches in an instance of the comparing where an image of the set of images does not meet the quality requirements: deleting the image of the images;
Li BASE is analogous art with respect to Shroff in view of the following: Peruch, Hofer, and Said because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of the following: Peruch, Hofer, and Said to wherein tracking: as the set of images are captured in rapid succession: comparing each image to definition degree; in an instance of the comparing where an image of the set of images does not meet the quality requirements: deleting the image of the images; providing, to the user, audio feedback that indicates a change in the movement of the user’s head regarding the region of interest being outside the field of view, the change being adapted to enable adjusting the user head relative to the object of interest. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
However, Shroff in view of the following: Peruch, Hofer, Said, and Li fail to teach the following: providing, to the user, notification that indicates a change in user location, the change being adapted to enable obtaining of a replacement image for the image of the images.
But Stathacopoulos does. Stathacopoulos teaches:
providing, to the user, notification that indicates a change in user location, the change being adapted to enable obtaining of a replacement image for the image of the images
“generate another notification for the user, changing the user's device position to capture the next frame of the scene.” (Stathacopoulos, [24])
Stathacopoulos teaches another notification that indicates a change in user's device position, the change being adapted to enable capturing of the next frame of the scene.
Stathacopoulos BASE is analogous art with respect to Shroff in view of the following: Peruch, Hofer, Said and Li because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of the following: Peruch, Hofer, Said and Li to wherein tracking: as the set of images are captured in rapid succession: compared each image of the set of images to compare each image to a comparison result; in an instance of the comparing where an image of the set of images does not meet the quality standard classifying them in second quality category: deleting the image of the images; providing, to the user, another notification audio feedback that indicates a change in user's device position, the change being adapted to enable capturing of the next frame of the scene. A person of ordinary skill in the art would do such in order to improve useability.
Claim(s) 14, 15, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shroff in view of the following: Peruch, Hofer, and Shin (US20230359271A1) hereinafter referenced as Shin.
Regarding claim 14, Shroff in view of Peruch and Hofer teaches the method of claim 13, and additionally teaches the following. Shroff teaches:
wherein the integrated sensing and interaction component further comprises: speakers; a microphone array; and a touch pad, a pair of cameras.
“ a speaker 127 to communicate to user 101 voice or sound signals… In some embodiments, smart glass 100 - 1 may include multiple sensors 121 such as IMUs, gyroscopes, microphones, and capacitive sensors configured as touch interfaces for the user.[28] … of a first forward-looking camera 315 - 1 … a second forward-looking camera 315 - 2 (hereinafter, first and second forward-looking cameras 315 - 1 and 315 - 2 will be referred to, hereinafter, as ‘cameras 315’ [34]”) (Shroff, [28, 34])
Shroff teaches the smart glasses comprises: two cameras; a speaker; microphones; and a touch interface.
Peruch continues and teaches:
Wherein the two image sensors are a pair of cameras;
“The first camera 102 and the second camera 104 include corresponding image sensors 102a and 104a” (Peruch, [39])
PNG
media_image1.png
657
882
media_image1.png
Greyscale
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer with the feature of Peruch to use at least two image sensors corresponding to the two cameras. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Shroff fails to teach speakers. But Shin does. Shin teaches:
Speakers
“the frame may include … interface devices (e.g., speakers, display, network adapter, etc.)” (Shin, [42])
Shin BASE is analogous art with respect to Shroff in view of Peruch and Hofer because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer wherein multiple sensors comprises: two cameras; speakers; microphones; and a touch interface. A person of ordinary skill in the art would do such in order to improve user experience.
Regarding claim 15, Shroff in view of Peruch, Hofer, and Shin teaches the method of claim 14, and additionally teaches the following. Shroff teaches:
wherein the integrated sensing and interaction component is adapted to: obtain the stereo images captured by the pair of cameras;
“of a first forward-looking camera 315 - 1 based on a region of interest for image capture, according to some embodiments. The scene 310 may be captured also by a second forward-looking camera 315 – 2” (Shroff, [34])
PNG
media_image2.png
477
692
media_image2.png
Greyscale
FIG 3. (Shroff)
Shroff teaches wherein smart glasses is adapted to: obtain the stereo image, via the multiple forward-looking cameras capturing the same scene which enables stereo imaging, from the pair of cameras;
and use at least the speakers to perform the action.
“a speaker 127 to communicate to user 101 voice or sound signals indicative of adjusting a gaze direction for improving the FOV of camera 115” (Shroff, 27)
Shroff teaches a speaker communicate to the user to adjust the gaze direction.
Peruch teaches:
at least partially process a first stereo image of the stereo image, as part of the capturing and processing of each of the stereo images, to obtain an image processing result, for the first stereo image;
“Referring to FIG. 3, in operation 302, the processor (e.g., the host processor 108 in FIG. 2A or the processor 210 working in concert with the controller 108 of FIG. 2B) controls the scanning device to capture an initial image. [60] … the initial image may be captured along an optical axis perpendicular to the ground plane that the object is resting on (e.g., a top-view of the object). [61] … Because this quick scan is used to initialize the process, issues of low coverage and under-sampling associated with this initial scan will be addressed and corrected through the process. [62]” (Peruch, [60-62])
Peruch teaches of capturing and processing the initial image as apart of the initialization process for scanning (reads on at least partially process a first stereo image of the stereo image, as part of the capturing and processing of each of the stereo images). The initialization process produces the initial scan to be used for further processing (reads on to obtain an image processing result, for the first stereo image).
identify an action to be performed based, at least in part, on the image processing result and a derived result from a remote entity, the derived result being based, at least in part, on the first stereo image and/or the image processing result of the first stereo image;
“in accordance with the one or more initial images to identify one or more next poses; controlling, by the processor, the camera to capture one or more additional images from at least one of the one or more next poses … and outputting the images captured by the camera to generate a three-dimensional model. [18] … image can be obtained by different methods … stereo camera systems [55] … the process of finalizing the model in operation 310 includes transmitting some partially processed or raw data to a remote processor to generate the finalized 3D model. [113]” (Peruch, [18, 55, 103])
Peruch teaches, in accordance with the one or more initial images, identifying one or more next poses and controlling the camera to capture additional images. The next poses are determined based on prior captured images, which implies processing those images to guide the subsequent capture. The remote processor uses the transmitted image data to generate the finalized 3D model.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch, Hofer, and Shin with the feature of Shroff and Peruch wherein the smart glasses is adapted to: obtain the stereo image, via the multiple forward-looking cameras capturing the same scene enabling stereo imaging, from the pair of cameras; at least partially process the stereo image to obtain an image processing result of a 3D model; identifying the next poses and controlling the camera to capture additional images based on prior image processing while also incorporating a derived result from a remote processor that uses transmitted image data to generate the final 3D model; and use at least a speaker to communicate to the user to adjust the gaze direction. A person of ordinary skill in the art would do such in order to improve image processing efficiency and user experience.
Hofer continues and teaches
stitching of each of the images
“If the user captures the second image when the camera is so aligned, there will presumably be adequate overlap between the two images (e.g., approximately 1/3 of the image width) to later join them together using an appropriate "stitching" program.” (Hofer, [3])
Hofer teaches stitching the overlap between the two images of a scene captured by user.
Hofer BASE is analogous art with respect to Shroff in view of Peruch because they are from the same field of endeavor, namely image capturing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch to create a method for obtaining information regarding a scene using smart glasses, the method comprising: identifying a region of interest in the scene as indicated by a user of a device; computing a guidance map based the one or more initial images identify next poses; actively guiding user movement via a dynamic positioned composite guide, based on a planned sequence of scene portions to be captured, to obtain set images of the scene, and to stitch the overlap between the two images of a scene captured by user. A person of ordinary skill in the art would do such in order to improve engagement, usability, and rewarding experiences.
Regarding claim 17, Shroff in view of Peruch, Hofer, and Shin teaches the method of claim 14, and additionally teaches the following. Peruch teaches:
wherein, each of the images of the pair of images of each of the stereo images are simultaneously being captured at different angles and/or positions with respect to the scene by the pair of cameras.
“the processor may control cameras 102 and 104 to capture images at substantially the same time and use the captured images to compute a depth image using image disparity techniques as described above [60] … stereoscopic depth camera system includes at least two cameras that are spaced apart from each other and rigidly mounted to a shared structure such as a rigid frame. The cameras are oriented in substantially the same direction … and have overlapping fields of view.” (Peruch, [60, 63])
Peruch teaches wherein the stereo image comprises a pair of images of the scene captured by at least two cameras spaced apart from each other with overlapping field of view at the same time.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch, Hofer, and Shin with the feature of Peruch wherein the stereo image comprises a pair of images of the scene captured by at least wo cameras spaced apart from each other with overlapping field of view. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shroff in view of the following: Peruch, Hofer, Shin, and Bleyer (US2021160441A1), hereinafter referenced as Bleyer
Regarding claim 16, Shroff in view of Peruch, Hofer, and Shin teaches the method of claim 14, and additionally teaches the following. Shroff teaches:
wherein the pair of cameras comprise lenses configured to: establish a camera line of sight that is converging to a line of sight of the user;
“in the eyepieces of the smart glass, two converging lines indicative of the gaze direction” (Shroff, [48])
Shroff teaches wherein the pair of cameras comprise lenses configured to: establish a camera line of sight that is converging to a line of sight of the user.
and establish a camera field of view that comprises the field of view of the user.
“Camera 515 faces forward towards the user FOV.” (Shroff [35])
Shroff teaches establishing a camera field of view that comprises the field of view of the user.
Shroff does not teach a camera line of sight that is parallel to a line of sight of the user. But Bleyer does. Bleyer teaches:
a camera line of sight that is parallel to a line of sight of the use
“the perspective appears as though those camera image(s) 315 were captured using cameras having parallel (or near parallel or at least corresponding to an alignment of human eyes) optical axes.” (Bleyer, [72])
Bleyer teaches a camera line of sight that is parallel to a line of sight of the user.
Bleyer BASE is analogous art with respect to Shroff in view of Peruch, Hofer, and Shin because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch, Hofer, and Shin to wherein the pair of cameras comprise lenses configured to: establish a camera line of sight that is parallel to a line of sight of the user; establish a camera field of view that comprises the field of view of the user. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable Shroff in view of Peruch, Hofer, and Wexler (US2020053262A1), hereinafter referenced as Wexler.
Claim 18 is rejected using the same rationale or bases as applied to claim 1 and the mentioned structure. Additionally, claim 18 recites the following structure as taught by Wexler.
a non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations
“In one embodiment, a non-transitory computer-readable medium for use in a system employing a wearable image sensor pairable with a mobile communications device, the computer readable medium containing instructions. The instructions, when executed by at least one processor” (Wexler, [08])
Wexler BASE is analogous art with respect to Shroff in view of Peruch and Hofer because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer to create a non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable Shroff in view of Peruch, Hofer, and Fisker (US8837026B2), hereinafter referenced as Fisker.
Claim 20 is rejected using the same rationale or bases as applied to claim 1 and the mentioned structure. Additionally, claim 20 recites the following structure as taught by Fisker.
A data processing system, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations
“a data processing system as described above for producing a 3D computer model of a physical object, … a central processing unit, a memory… stored therein data representing sequences of instructions which when executed cause the method of the invention to be performed.” (Fisker, [47])
Fisker BASE is analogous art with respect to Shroff in view of Peruch and Hofer because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Shroff in view of Peruch and Hofer to a data processing system, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations. A person of ordinary skill in the art would do such in order to improve image processing efficiency.
Applicant Argument
Applicant argues “none of these cited prior art references actually teach the obtaining of stereo images (which are a very specific type of images, and also a term of art in the field of imaging) and the
stitching of the pair of images included in each stereo image”. Additionally, applicant submits that “none of these cited prior art references teach the simultaneous capturing of pair of images making up a stereo image using at least two image sensors”. Applicant amended claims accordingly to reflect.
Response to Argument
The prior art does teach stereo images and stitching of the pair of images included in each stereo image. Peruch teaches obtaining stereo images captured by image sensors. Peruch explains “The depth camera may be a stereoscopic depth camera including: a first camera having a first field of view with a first optical axis; and a second camera spaced apart from the first camera and having a second field of view overlapping the first field of view and having a second optical axis substantially parallel to the first optical axis.” This is the stereo images the applicant argued. Peruch describes two images of the same scene captured from two spaced apart parallel axis viewpoint with overlapping coverage. Peruch goes further and cites an academic reference, R. Szeliski. “Computer Vision: Algorithms and Applications”, Springer, 2010 pp. 467 et seq, for this concept confirming that this is conventional and well-understood and not a novel concept.
Additionally, Hofer teaches stitching captured overlapping images. The idea of stitching overlapping images is a conventional and well-understood image combination technology. Given Peruch’s overlapping stereo image pairs, a person of ordinary skill in the art would be aware of Hofer’s teaching to stitch captured overlapping images to create a single well combined image.
Therefore, the amended claims are rejected.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUNE NGUYEN whose telephone number is (571)272-8919. The examiner can normally be reached M-TH 7:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUNE NGOC NGUYEN/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
