Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Arguments
Applicant’s arguments, filed July 3, 2025, with respect to how the newly amended claim features differ from the prior art cited in the last office have been fully considered. These arguments are found to be persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in this office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 6, 9, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Kachach et al. (NPL Doc, “Virtual Tour: An Immersive Low Cost Telepresence System”) in view of Klein (Patent No. US 6,987,545 B1) in further view of Chen et al. (Patent No. 11,778,427 B2).
As per claim 1, Kachach teaches the claimed:
1. A computer implemented method, comprising:
obtaining a previously captured image data of a virtual tour of a location (Kachach in the abstract “In this work we present Virtual Tour, a low cost telepresence system that provides the possibility of visiting a remote place … the remote environment is captured, stitched and encoded by the 360° camera. Resulting video and audio are streamed using a custom low latency streaming solution” and in section 3, 1st paragraph “… These tests included different scenarios such as: 1) attend a meeting remotely, 2) indoor virtual tour in the office … On this configuration we can reach an end-to-end delay under 400 milliseconds which is enough to have a good immersion and to maintain a real-time conversation”.
In this instance, the obtained previously captured image is a video frame in the video stream captured by the 360° camera providing a virtual tour of the indoor office/laboratory captured at an earlier time. It is obtained previously because the video frames are captured over time, e.g. it may be an image that is 400 milliseconds or older. Thus, it is a video frame captured before the most recent one),
receiving real-time image data of a target object located within a field of view area of a camera, wherein the target object is located in a same location where the previously captured image data was captured (Kachach in figure 1 and in the abstract “In this work we present Virtual Tour, a low cost telepresence system that provides the possibility of visiting a remote place … the remote environment is captured, stitched and encoded by the 360° camera. Resulting video and audio are streamed using a custom low latency streaming solution” and in section 3, 1st paragraph “… These tests included different scenarios such as: 1) attend a meeting remotely, 2) indoor virtual tour in the office … On this configuration we can reach an end-to-end delay under 400 milliseconds which is enough to have a good immersion and to maintain a real-time conversation”.
In this instance, real-time image data is received of a target object (another person located in the remotely located office/ laboratory), e.g. as shown in figure 1. The right side of figure 1 shows that the target object (another person located in the remotely located office/ laboratory) is located within a field of view area of the camera located in the remotely located office for the virtual office tour. In figure 1 on the left side, the “Remote Participant” receives the real-time image data. The images received are in real-time using the video stream because the Kachach reference indicates that the remote participant can maintain a real-time conversation during the indoor virtual tour in the office.
The target object (another person located in the remotely located office) is in the same location where the previously captured image data was captured (the previous video frame from the video stream, e.g. 400 milliseconds earlier) when the person is standing still for a moment),
receiving real-time acoustic data captured by an audio sensor in the location (Kachach in section 2.1, 1st paragraph “… Due to some technical limitations on the camera side, conference audio is captured/reproduced by means of the Jabra speaker which provides high quality audio and echo-cancelling capabilities.” Also, please see Kachach in the abstract “ …Resulting video and audio are streamed using a custom low latency streaming solution … In addition, conference capabilities of the proposed system boosts the telepresence feeling since the user can have a real-time conversation with people on the remote scenario”.
Kachach in figure 1 on the left side shows that the user receives real-time acoustic data captured by an audio sensor located in the remote office (the location) during the indoor office tour (the location)).
wherein the acoustic data is captured by the audio sensor in a real world (Kachach in section 2.1, 1st paragraph “… Due to some technical limitations on the camera side, conference audio is captured/reproduced by means of the Jabra speaker which provides high quality audio and echo-cancelling capabilities.”. Kachach in figure 1 on the right side shows that the audio (acoustic data) is captured by the Jabra speaker located in the real-world remotely located office during the indoor office tour);
Kachach alone does not explicitly teach the remaining claim limitations.
However, Kachach in combination with Klein teaches the claimed:
overlaying the real-time image data of the target object on a corresponding area of the target object in a virtual tour view for forming a mixed image virtual tour (Kachach in figure 1 on the left side, it shows that the user (Remote Participant) sees the real-time image data of the target object (remotely located person in the office) on a head-mounted display (HMD) headset. Also, please refer to the abstract and in section 3, 1st paragraph in Kachach. This target object (remotely located person in the office) is displayed on a corresponding area of the target object in a virtual office tour view for forming a mixed image virtual tour.
While Kachach shows these features, Kachach does not clearly show that overlaying is occurring per se. Klein teaches that this feature was known in the art, e.g. please see Klein in col 2, lines 19-23: “the memory stores a current video frame and a previous video frame in the same location in the memory, allowing the current video frame to be written over the previous video frame.” In this instance, the current video frame writing over the previous video frame corresponds to the claimed “overlaying the real-time image data of the target object on a corresponding area of the target object”. When this overlaying of Klein is applied to the video frames of Kachach, the claimed features are taught because video frames in Kachach contain the target object (person in the indoor office) appearing in the virtual indoor office tour (virtual tour view). These references are combined by using Klein’s overlay to write over a previous video frame in order to provide a real-time updated video stream display of the virtual tour for the remotely located user),
and all overlayed image data is captured at a same time point (As mentioned above, Klein in col 2, lines 19-23 teaches of overlaying image data when updating a previous video frame with a current video frame. Kachach teaches of capturing a current video frame in figure 1 and in the abstract “In this work we present Virtual Tour, a low cost telepresence system that provides the possibility of visiting a remote place … the remote environment is captured, stitched and encoded by the 360° camera. Resulting video and audio are streamed using a custom low latency streaming solution” and Kachach in section 3, 1st paragraph “… These tests included different scenarios such as: 1) attend a meeting remotely, 2) indoor virtual tour in the office … On this configuration we can reach an end-to-end delay under 400 milliseconds which is enough to have a good immersion and to maintain a real-time conversation”.
In this instance, the current video frame is part of the real-time image data captured by the camera. The current video frame represents the image data captured at a same time point by the camera placed in the real-world scene (in the indoor office on the tripod as shown in figure 1 of Kachach). The current video frame becomes the overlayed image data when Kachach is combined with Klein).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to overlay the image data as taught by Klein with the system of Kachach in order to use the same memory area to update the video display over time (Klein in col 2, lines 19-22). This may simplify implementation of the memory setup used for displaying the video frame data.
Kachach and Klein in combination with Chen teaches the claimed:
playing the real-time acoustic data when a user moving along the mixed image virtual tour exceeds a predetermined distance from a position of the target object (Kachach teaches of playing real-time audio (acoustic data) for a user moving along a mixed image with virtual tour, e.g. please see Kachach in figure 1 and in section 2.1, 1st paragraph “… Due to some technical limitations on the camera side, conference audio is captured/reproduced by means of the Jabra speaker which provides high quality audio and echo-cancelling capabilities” and Kachach in section 3, 1st paragraph “… These tests included different scenarios such as: 1) attend a meeting remotely, 2) indoor virtual tour in the office … On this configuration we can reach an end-to-end delay under 400 milliseconds which is enough to have a good immersion and to maintain a real-time conversation”.
While Kachach teaches of these features, Kachach is silent about the claimed “when a user moving along the mixed image virtual tour exceeds a predetermined distance from a position of the target object” per se. Chen teaches that this feature was known in the art, e.g. please see Chen in their claim 1 language which recites: “determining whether a distance between the first wearable electronic device and the at least one second wearable electronic device is larger than a distance threshold … to start a walkie-talkie conversation with the at least one second wearable electronic device when the distance is larger than the distance threshold, the first wearable electronic device and the at least one second wearable electronic device being controlled in the audio playing mode to play an audio packet stream”.
The claimed feature is taught when this distance determination of Chen is incorporated into the audio conversation in figure 1 of Kachach. When these references are combined, the microphone on the tripod in the indoor office in figure 1 of Chen corresponds to a position of the target object because it is recording the voice of the person standing in the office. The user wearing the head-mounted display on the left side of figure 1 in Chen corresponds to the claimed “user” because this is the person watching the video stream of the indoor office virtual tour).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the predetermined distance as taught by Chen with the system of Kachach as modified by Klein. This feature from Chen is useful for situations where the user walks to eventually be located close to the people in the indoor office tour. In such a scenario, Chen is able to turn off the audio playing for situations where the two people (with respective electronic devices) are close enough to hear and talk to each in-person without any audio playing over the network/Internet. Thus, Chen’s feature is useful because the audio conversation only plays when the two people (with respective electronic devices) are located faraway enough from each other for the audio playing to be useful for having a conversation.
As per claim 2, Kachach teaches the claimed:
2. The computer implemented method according to claim 1, wherein the previously captured image data is static image data of a real environment of the location (Kachach in the abstract “In this work we present Virtual Tour, a low cost telepresence system that provides the possibility of visiting a remote place … the remote environment is captured, stitched and encoded by the 360° camera. Resulting video and audio are streamed using a custom low latency streaming solution” and in section 3, 1st paragraph “… These tests included different scenarios such as: 1) attend a meeting remotely, 2) indoor virtual tour in the office … On this configuration we can reach an end-to-end delay under 400 milliseconds which is enough to have a good immersion and to maintain a real-time conversation”.
In this instance, the obtained previously captured image is a video frame in the video stream captured by the 360° camera providing a virtual tour of the indoor office. It is obtained previously because the video frames are captured over time, e.g. it may be an image that is 400 milliseconds or older. This video frame is static image data of a real environment of the location because it is a snapshot in-time taken of the indoor real office environment by the camera).
As per claim 4, Kachach teaches the claimed:
4. The computer implemented method according to claim 1, wherein the previously captured image data is a ready-constructed virtual tour (As mentioned above for claims 1 and 2, the obtained previously captured image is a video frame in the video stream captured by the 360° camera providing a virtual tour of the indoor office/ laboratory (e.g. please see figure 1, the abstract, and in section 3, 1st paragraph of Kachach). It is obtained previously because the video frames are captured over time, e.g. it may be an image that is 400 milliseconds or older. This previously captured image data (previously capture video frame) is a ready-constructed virtual tour of the indoor office viewable by the remote user as shown in figure 1 of Kachach).
As per claim 6, Kachach teaches the claimed:
6. The computer implemented method according to claim 1, wherein the real-time image data is a real-time video stream or frequently captured real-time images (Kachach in figure 1 and in the abstract “In this work we present Virtual Tour, a low cost telepresence system that provides the possibility of visiting a remote place … the remote environment is captured, stitched and encoded by the 360° camera. Resulting video and audio are streamed using a custom low latency streaming solution” and in section 3, 1st paragraph “… These tests included different scenarios such as: 1) attend a meeting remotely, 2) indoor virtual tour in the office … On this configuration we can reach an end-to-end delay under 400 milliseconds which is enough to have a good immersion and to maintain a real-time conversation”.
In this instance, the real-time image data is a real-time video stream or frequently captured real-time images. The images received are in real-time because the Kachach reference indicates that the remote participant can maintain a real-time conversation during the indoor virtual tour in the office).
As per claim 9, this claim is similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale. The system of Kachach would have to have some type of data processing unit present in order to function and run on a computer as described by the reference.
As per claim 13, this claim is similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale. The system of Kachach would have to have some type of non-transitory computer readable medium present in order to function and run on a computer as described by the reference.
Claims 3, 5, 7, and 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Kachach in view of Klein in further view of Chen and Cosgun et al. (NPL Doc, “Autonomous Person Following for Telepresence Robots”).
As per claim 3, Kachach alone does not explicitly teach the claimed limitations.
However, Kachach in combination with Cosgun teaches the claimed:
3. The computer implemented method according to claim 1, wherein the previously captured image data comprises images captured by a moving platform moving through the location, and wherein the previously captured image data is obtained from a camera, a memory or a cloud storage, and wherein the method further comprises: constructing the previously captured image data as a virtual tour by a data processing unit (Kachach in figure 1 shows that the image data (including previously captured image data) is captured from a tripod platform with a camera located at the location. Also, a virtual tour of an indoor office is constructed in Kachach using this image data. Kachach would have to have some type of data processing unit present in order to perform the steps as described in their reference.
Kachach, however, is silent about a moving platform per se. Cosgun teaches that this feature was known in the art, e.g. please see Cosgun in figure 1 where the platform capturing the image data is mobile. Previously captured image data is captured over time, e.g. during the real-time video stream captured by the camera older video frames become the previously captured image data).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the mobile platform as taught by Cosgun with the system of Kachach as modified by Klein and Chen in order to allow the camera to move with the other person or tour guide as they move throughout the office or other tour location. Kachach also suggests using a mobile platform with their own system at the end of section 4 in Kachach.
As per claim 5, Kachach alone does not explicitly teach the claimed limitations.
However, Kachach and Klein in combination with Cosgun teaches the claimed:
5. The computer implemented method according to claim 1, wherein the method further comprises: obtaining additional information about the location, and overlaying the additional information on a mixed image virtual tour view (As mentioned above for claim 1, Kachach teaches of capturing real-time image data of the mixed image virtual tour view for a remote user and Klein teaches of overlaying. Kachach is silent about obtaining additional information per se. Cosgun teaches this feature. Cosgun in figure 2 teaches of obtaining additional information about the location such as a depth image and a laser scan of the planned path. This information is added into the mixed image virtual tour view in figure 2 of Cosgun. Klein’s overlay may be used with the additional information of Cosgun to update the video display frames over time when these references are combined together).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the overlaying as taught by Klein with the system of Kachach. The motivation of claim 1 is incorporated herein.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the additional information as taught by Cosgun with the system of Kachach as modified by Klein and Chen. The additional depth camera data or laser scan data may help provide better 3D understanding of the spatial area in the real-world where the tour is located. This could help locate nearby people as well as other structures or features in the real-world area around the cameras.
As per claim 7, Kachach alone does not explicitly teach the claimed limitations.
However, Kachach and Klein in combination with Cosgun teaches the claimed:
7. The computer implemented method according to claim 1, wherein the method further comprises: capturing real-time image data configured to be overlayed on areas of the virtual tour view by more than one camera (As mentioned above for claim 1, Kachach teaches of capturing real-time image data of the virtual tour and Klein teaches of overlaying. Kachach is silent about using more than one camera per se. Cosgun teaches this feature. For example, please see Cosgun in figure 2 where it shows that a wide-lens camera is used as well as a depth camera from a Kinect device. Thus, the captured real-time image data is configured to be overlayed on areas of the virtual tour view by more than one camera).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the overlaying as taught by Klein with the system of Kachach. The motivation of claim 1 is incorporated herein.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use more than 1 camera as taught by Cosgun with the system of Kachach as modified by Klein. The additional depth camera may help provide better 3D understanding of the spatial area in the real-world where the tour is located. This could help locate nearby people as well as other structures or features in the real-world area around the cameras.
As per claims 10-11, these claims are similar in scope to limitations recited in claims 5 and 7, respectively, and thus are rejected under the same rationale.
Claims 8 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Kachach in view of Klein in further view of Chen and Reddy et al. (Pub No. US 2016/0260253 A1).
As per claim 8, Kachach alone does not explicitly teach the claimed limitations.
However, Kachach in combination with Reddy teaches the claimed:
8. The computer implemented method according to claim 1, wherein a time point of views of the mixed image virtual tour is changeable to a previous time point (As mentioned above for claim 1, Kachach in figure 1 and in the abstract teaches of displaying a mixed image virtual tour. Kachach is silent about a time point of views being changeable to a previous time point. Reddy teaches that this feature was known in the art, e.g. please see Reddy in [0080] “In particular embodiments, the point corresponding to the selected thumbnail is prior to the current point according to the order of points or the images, the virtual tour in which the image corresponding to the current point has been displayed may be rewound until the image corresponding to the selected thumbnail is displayed in the virtual tour.” This is also shown in figure 8 of Reddy as well. The claimed feature is taught when these selectable thumbnails of Reddy are incorporated into the virtual tour in Kachach).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for a time point of views being changeable to a previous time point as taught by Reddy with the system of Kachach as modified by Klein and Chen. This provides the user with more control and flexibility over the display presentation because it allows the user to go back to view earlier portions of the virtual tour, e.g. for the user’s own review.
As per claim 12, this claim is similar in scope to limitations recited in claim 8, and thus is rejected under the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL F HAJNIK whose telephone number is (571) 272-7642. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616