DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims, the Examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the Examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/16/2026, was filed after the mailing of a First Office Action on the Merits but before the close of prosecution. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. The current application claims foreign priority to JP2024-028399 which has an earlier filed date of 02/28/2024. Examiner has checked and verified that the subject matter of the instant application is supported by the foreign document, and as such, the earlier filed date of 02/28/2024 is granted.
Response to Amendment/Argument
The amendments filed on 08/03/2026 have been entered. Claims 1-4 and 6-11 remain pending in the application. Examiner notes that the previous 35 U.S.C. 112(b) section regarding claim 3 has been removed as the limitations causing the issue have been amended out.
Claim Rejections - 35 USC § 101
Examiner notes that the current independent claims are not rejected under 35 U.S.C. 101 because they recite a homography process which is not something that can reasonably be performed in the human mind, especially while a vehicle is moving. This process requires numerous matrix conversions of individual pixels of an image which can, depending on the resolution of the image, be in the millions to hundreds of millions (megapixel scale, etc.) which is simply not something that can be reasonably performed in the human mind.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1 and 6-9 are rejected under 35 U.S.C. 103 as being obvious over Kelly et al., US 20110066262 A1, herein referred to as Kelly, and in view of Fujita et al., US 20150130942 A1, herein referred to as Fujita.
Regarding claim 1, Kelly discloses a remote operator terminal used by a remote operator for a remote operation of a moving body (Figs. 4, 6-7, Paragraph 0047; the remote system may be used for remote operation of a vehicle), a first image is an image captured at a first timing by a camera mounted on the moving body (Figs. 4, 6-7, Paragraphs 0047-0052; cameras may acquire data in real-time, cameras may be attached to the remote apparatus which can be a vehicle, this data being acquired can be considered as being acquired at a first timing), a first point of view is defined by a combination of a position and a viewing direction of the camera at the first timing (Figs. 4, 6-7, Paragraphs 0047-0052; each camera may have a given field of view based on its current position and viewing direction), a second point of view is defined by a combination of a predicted position and a predicted viewing direction of the camera at a second timing later than the first timing (Figs. 4, 6-7, Paragraphs 0047-0052, 0066-0069; a synthetic point of view may be created through a predicted camera points of view, this prediction may be based on positioning, orientation, etc.), a homography process converts the first image viewed from the first point of view into a second image viewed from the second point of view based on perspective projection transformation using a projection plane associated with the camera (Figs. 4, 6-7, Paragraphs 0047-0052, 0066-0070; a homography process may be used to produce a synthetic image frame, this second image frame has a second point of view, examiner notes that homography is a term of art and is well defined as utilizing a common projection plane between two perspectives), acquire a plurality of first images respectively captured at the first timing by a plurality of cameras mounted on the moving body (Figs. 4, 6-7, Paragraphs 0047-0052, 0066-0070; multiple cameras may be used for capturing images at a first time), acquire a combination of a plurality of second images by applying the homography process using a common projection plane to a combination of the plurality of first images (Figs. 4, 6-7, Paragraphs 0047-0052, 0066-0070; multiple synthetic images may be obtained through a homography process which utilizes a common projection plane), and display the plurality of second images on a display of the remote operator terminal (Figs. 4, 6-7, Paragraphs 0047-0052, 0061, 0066-0070; the synthetic images may be output to a user through a video screen), but fails to disclose wherein installation directions of the plurality of cameras are different from each other and displaying the plurality of second images side-by-side on one or more screens of a display of the remote operator terminal.
However, Fujita, in an analogous field of endeavor, teaches wherein installation directions of the plurality of cameras are different from each other (Fig. 2; the vehicle may have cameras mounted around the vehicle, each having a different installation direction and field of view), and displaying the plurality of second images side-by-side on one or more screens of a display of the remote operator terminal (Fig. 12, Paragraph 0117; images may be displayed next to each other; ‘side-by-side’ under BRI can mean that any side of one image is next to another side of another image). Therefore, from the teaching of Fujita, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified, with a reasonable expectation for success, the remote system of Kelly to include wherein installation directions of the plurality of cameras are different from each other and displaying the plurality of second images side-by-side on one or more screens of a display of the remote operator terminal, as taught/suggested by Fujita. The motivation to do so would be to ensure that the cameras obtaining the images are able to obtain as much of the environment (or object of interest) as possible. Additionally, displaying the homography processed images close together can allow for ease of viewing to the remote operator. This can reduce the need to move windows around and can reduce how often an operator needs to look back and forth, which can reduce eye strain.
Regarding claim 6, Kelly in view of Fujita renders obvious all the limitations of claim 1. Kelly further discloses estimating a direction and an amount of movement of the moving body in a period from the first timing to the second timing (Figs. 4, 6-7, Paragraphs 0047-0053, 0061, 0066-0070, 0080-0082; the homography process may be used to generate synthetic images based on first images obtained by a real camera, the system may predict the viewing direction of the synthetic camera and generate synthetic images based on predicted locations of the apparatus (which can be a vehicle)), calculating a difference between the first point of view and the second point of view based on the direction and the amount of movement of the moving body (Figs. 4, 6-7, Paragraphs 0047-0053, 0061, 0066-0070, 0080-0082; the homography process used to generate the synthetic images are based on movement of the apparatus which includes a given direction (0081 discloses movement along a road which is a direction), the predicted locations from movement are used to determine differences between synthetic images, which includes differing points of view), and converting the first image viewed from the first point of view into the second image viewed from the second point of view based on the difference between the first point of view and the second point of view (Figs. 4, 6-7, Paragraphs 0047-0053, 0061, 0066-0070, 0080-0082; the synthetic images obtained through homography and are generated based on differences between predicted positions of the apparatus, these synthetic images are based off of real camera images obtained through the one or more real cameras through the homography process which is a conversion between real and synthetic images).
Regarding claim 7, Kelly in view of Fujita renders obvious all the limitations of claim 1. Kelly further discloses set at least a part of a communication delay time between the moving body and the remote operator terminal as a delay compensation time (Figs. 4, 6-7, Paragraphs 0047-0053, 0061, 0066-0070, 0080-0082; latency may be utilized in generating the synthetic images through homography, this latency can be considered the delay compensation time as the system utilizes it when generating the synthetic images) and set the second timing to be later than the first timing by the delay compensation time (Figs. 4, 6-7, Paragraphs 0047-0053, 0061, 0066-0070, 0080-0082; sequential synthetic images may be generated based on a latency which result in the sequential synthetic images to be at a later time than those images taken by the real camera).
Regarding claim 8, the claim limitations are similar to those in claim 1 and are rejected using the same rationale as seen above in claim 1.
Regarding claim 9, the claim limitations are similar to those in claim 1 and are rejected using the same rationale as seen above in claim 1.
Claim 10 is rejected under 35 U.S.C. 103 as being obvious over Kelly, in view of Fujita, and further in view of Chan et al., US 10192287 B2, herein referred to as Chan.
Regarding claim 10, Kelly in view of Fujita renders obvious all the limitations of claim 1. Kelly further discloses display the plurality of second images on a display of the remote operator terminal (Figs. 4, 6-7, Paragraphs 0047-0052, 0061, 0066-0070; the synthetic images may be output to a user through a video screen), but fails to disclose wherein the processing circuitry is configured to display the plurality of second images side-by-side such that boundary lines of adjacent second images coincide with each other without any overlap and gap between the adjacent second images. However, Chan, in an analogous field of endeavor, teaches wherein the processing circuitry is configured to display the plurality of second images side-by-side such that boundary lines of adjacent second images coincide with each other without any overlap and gap between the adjacent second images (Fig. 6, Col. 7 lines 43-52, Col. 8 line 64 - Col. 9 line 32 ; separate images may be fused together to create one large image without gaps or overlap; this fused image may then be displayed). Therefore, from the teaching of Chan, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified, with a reasonable expectation for success, the remote system of Kelly and Fujita to include wherein the processing circuitry is configured to display the plurality of second images side-by-side such that boundary lines of adjacent second images coincide with each other without any overlap and gap between the adjacent second images, as taught/suggested by Chan. The motivation to do so would be to display an image that is pleasing to look at. Additionally, having no gaps between images allows for better viewing and can reduce eye strain.
Claim 2 is rejected under 35 U.S.C. 103 as being obvious over Kelly, in view of Fujita, and further in view of Brudnak, US 20190058853 A1, herein referred to as Brudnak.
Regarding claim 2, Kelly in view of Fujita renders obvious all the limitations of claim 1. Kelly further discloses a remote operator (Figs. 4, 6-7, Paragraph 0047; a remote operator may be used for controlling the remote apparatus) and multiple cameras (Figs. 4, 6-7, Paragraphs 0047-0052, 0066-0070; multiple cameras may be utilized), but fails to disclose select one of the plurality of cameras as a reference camera, and set the projection plane associated with the reference camera as the common projection plane. However, Brudnak, in an analogous field of endeavor, teaches select one of the plurality of cameras as a reference camera (at least Paragraphs 0046, 0054; a camera may be utilized for setting the reference planes for image projection; utilizing this camera can be considered as selecting a main camera as all projections and references are based on this camera), and set the projection plane associated with the reference camera as the common projection plane (at least Paragraphs 0046, 0054; a projection plane may be associated with the camera which is used as a common plane for all projections). Therefore, from the teaching of Brudnak, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified, with a reasonable expectation for success, the remote system of Kelly and Fujita to include select one of the plurality of cameras as a reference camera, and set the projection plane associated with the reference camera as the common projection plane, as taught/suggested by Brudnak. The motivation to do so would be to define the reference plane for the homography process. This can ensure that projections are based on a known baseline reference, which can lead to increased accuracy and efficiency in the system.
Claims 3-4 are rejected under 35 U.S.C. 103 as being obvious over Kelly, in view of Fujita, in view of Brudnak, and further in view of Otani et al., US 20210381197 A1, herein referred to as Otani.
Regarding claim 3, Kelly in view of Fujita and Brudnak renders obvious all the limitations of claim 2. Kelly further discloses multiple cameras with different field of views (Figs. 4, 6-7, Paragraphs 0047-0052, 0066-0070; multiple cameras may be utilized that have different field of views), but fails to disclose estimate a gaze direction of the remote operator, and select one camera corresponding to the gaze direction as the reference camera. However, Otani, in an analogous field of endeavor, teaches estimate a gaze direction of the remote operator (at least Paragraphs 0086-0097; a user’s direction of view may be determined based on their posture), and dynamically select one corresponding to the gaze direction among the plurality of cameras as the reference camera (at least Paragraphs 0086-0097; a camera of a plurality of cameras may be selected based on the user’s estimated gaze direction). Therefore, from the teaching of Otani, it would have been obvious to one of ordinary skill in the art before the effective filing date to have further modified, with a reasonable expectation for success, the remote system of Kelly, Fujita, and Brudnak to include estimate a gaze direction of the remote operator, and dynamically select one corresponding to the gaze direction among the plurality of cameras as the reference camera, as taught/suggested by Otani. The motivation to do so would be to provide relevant view points for a user based on their gaze direction. This can help the user better understand the environment by providing a projected view that is most aligned with their own view. This can lead to increased accuracy in remote vehicle operations as the environment is better presented to the user.
Regarding claim 4, Kelly in view Fujita, Brudnak, and further in view of Otani renders obvious all the limitations of claim 3. Kelly further discloses multiple cameras with different field of views (Figs. 4, 6-7, Paragraphs 0047-0052, 0066-0070; multiple cameras may be utilized that have different field of views), but fails to disclose estimate the gaze direction of the remote operator based on a direction of a line of sight of the remote operator or a steering angle of a steering operation performed by the remote operator. However, Otani teaches estimate the gaze direction of the remote operator based on a direction of a line of sight of the remote operator or a steering angle of a steering operation performed by the remote operator (at least Paragraphs 0086-0097; a camera of a plurality of cameras may be selected based on the user’s line of sight which is based on their posture). Therefore, from the teaching of Otani, it would have been obvious to one of ordinary skill in the art before the effective filing date to have further modified, with a reasonable expectation for success, the remote system of Kelly, Fujita, Brudnak, and Otani to include estimate the gaze direction of the remote operator based on a direction of a line of sight of the remote operator or a steering angle of a steering operation performed by the remote operator, as taught/suggested by Otani. The motivation to do so would be to provide relevant view points for a user based on their gaze direction/line of sight. This can help the user better understand the environment by providing a projected view that is most aligned with their own view. This can lead to increased accuracy in remote vehicle operations as the environment is better presented to the user.
Potentially Allowable Subject Matter
Claim 11 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 11, the examiner has performed a thorough search and has not found a piece of prior art, either alone or in combination with other prior art, that discloses, teaches, suggests, or renders obvious the claim limitations. The closest prior art combination, US 210110066262 by Kelly and US 20150130942 by Fujita, teaches all the limitations of claim 1, specifically including acquiring a plurality of first images respectively captured at the first timing by a plurality of cameras mounted on the moving body, wherein installation directions of the plurality of cameras are different from each other and acquiring a combination of a plurality of second images by applying the homography process using a common projection plane to a combination of the plurality of first images. However, neither reference teaches wherein the processing circuitry is configured to concatenate the plurality of first images to generate one large first image, and to apply the homography process using the common projection plane to the one large first image to acquire one large second image corresponding to the combination of the plurality of second images. This feature is novel in that it sets up an image collection in a certain spatial orientation before applying a homography process utilizing a common projection plane. Although Kelly discloses a homography process applied to first images to generate second images using a common projection plane, neither Kelly nor Fujita teach the first images first being concatenated beforehand to create a large first image, which may allow for better viewing and higher accuracy of the final generated large second image.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 and 8-9 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER ALLEN BUKSA whose telephone number is (571)272-5346. The examiner can normally be reached M-F 7:30 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached at (571) 272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.B./Examiner, Art Unit 3658
/THOMAS E WORDEN/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights