DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant(s) Response to Official Action
The response filed on 03/27/2026 has been entered and made of record.
Response to Arguments/Amendments
Presented arguments have been fully considered but are held unpersuasive. Examiner’s response to the presented arguments follows below.
Claim Rejections - 35 USC § 103
Summary of Arguments:
Regarding claims 1, 4, 9 the Applicant argues Otani in view of Kelly in further view of Kai:
“Otani merely considers the "overlay display (pop-up display)" and does not consider the "side-by-side display" at all. Therefore, the combination of Otani and Kelly does not result in the claimed invention. The above-mentioned technical problem occurs only when the "homography" and the "side-by-side display" are combined. In the case of the overlay display (pop-up display) of Otani, the technical problem never occurs. That is to say, there is no way that Otani recognizes the technical problem. Kelly also does not recognize the technical problem. Given the lack of recognition of the technical problem, there is no way to reach the claimed invention from Otani and Kelly. The other cited references, including, Kai do not address the deficiencies of Otani and/or Kelly.” [Remarks: Pages 8-9]
Regarding claims 3 & 5, the Applicant argues:
That the claims were previously objected to provided that the rejections under 35 U.S.C. 112(b) are overcome. [Remarks: Page 9]
Examiner’s Response:
Regarding claims 1, 4, 9, the Examiner contends:
Applicant's arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references. Further, the presented arguments have been fully considered, but are rendered moot since the amended claims lack written description support. Please see the updated rejection below for details.
Regarding claims 3 & 5, the Examiner contends:
Aside from the amended claims lacking written description support, the objected claims were never introduced into the independent claims, therefore the argument is moot.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-9 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
The original independent claims 1, 4 and 9 utilized broad “plurality” language, reciting the processing circuitry's configuration to “acquire a plurality of second images by applying the homography process to the plurality of first images, respectively”. This original limitation possessed adequate written description support within the specification, which explicitly describes acquiring “a plurality of first images IMG1 (IMG1-F, IMG1-L, and IMG1-R) respectively captured by the plurality of cameras” and acquiring “a plurality of second images IMG2 (IMG2-F, IMG2-L, IMG2-R)” by applying the homography process. However, the amended claims 1, 4 and 9 replace this supported limitation with a strict mathematical iteration, specifically reciting:
“… acquire n first images respectively captured at a first timing by n cameras mounted on the moving body, n being an integer equal to or greater than 2 …”.
“… a i-th (i = 1 to n) first image of the n first images …”.
“… acquire the i-th (i = 1 to n) second image by applying the homography process to the i-th first image to acquire n second images …”.
The specification fails to provide written description support for this newly claimed mathematical indexing scheme:
Unsupported Use of Variable n: The specification does not use the variable n (or n greater than or equal to 2) to define the number of cameras or the boundary of an image processing loop. The specification only introduces a similar variable, capital N, to represent a completely different element: “N virtual points are virtually set in the three dimensional world coordinate system” where “N is an integer of 4 or more”.
Unsupported Use of Index i: The specification entirely lacks the use of an i-th index to track or map individual cameras, viewpoints, and output images sequentially (i = 1 to n). Instead, the specification distinctly relies on directional labels (e.g., F for Front, L for Left, R for Right) to map the relationships between specific cameras (CAM-F) and their respective projection planes (P-F) and images (IMG1-F, IMG2-F).
While the specification supports the general concept of applying a homography process independently to multiple cameras, it does not reasonably convey to one skilled in the art that the inventor was in possession of the strictly defined mathematical iteration and index formulation (i = 1 to n) recited in the amended claims at the time of filing. Therefore, the amendments introduce new matter, rendering claims 1, 4 and 9 failing to comply with the written description requirement of 35 U.S.C. 112(a).
For the purposes of examination, the limitations are interpreted as the originally filed claims without the U.S.C. 112(b) rejections that were present in the previous office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6-9 are rejected under 35 U.S.C. 103 as being unpatentable over Otani et al., hereinafter referred to as Otani (US 2021/0381197 A1) in view of Kelly et al., hereinafter referred to as Kelly (US 2011/0066262 A1).
As per claim 1, Otani discloses a remote operator terminal used by a remote operator for a remote operation of a moving body (Otani: Abstract.),
the remote operator terminal comprising processing circuitry configured to acquire n first images respectively captured at a first timing by n cameras mounted on the moving body (Otani: Para. [0032] discloses “The plurality of cameras 11 to 15 [claimed n cameras] include a main camera 11 (an example of an image acquiring device) and a plurality of different viewpoint cameras 12 to 15” and Para. [0035] discloses “a plurality of images [claimed n first images] acquired by the plurality of cameras 11 to 15”), n being an integer equal to or greater than 2 (Otani: Para. [0034] discloses “The plurality of different viewpoint cameras 12 to 15 [claimed n being an integer equal to or greater than 2] have their respective visual fields different from the visual field of the main camera 11”), wherein
a i-th (i = 1 to n) first image of the n first images is an image captured at the first timing by a i-th camera of the n cameras-mounted on the moving body (Otani: Para. [0033] discloses “The main camera 11 [claimed i-th camera] acquires a main image [claimed i-th first image] that is an image of a work area of the hydraulic excavator 100”.),
a i-th first point of view is defined by a combination of a position and a viewing direction of the i-th camera at the first timing (Otani: Para. [0033] discloses “The main camera 11 is set, for example, at a position corresponding in height to the eyes of the operator sitting in the seat in the operator's cab, i.e., the cab 7, and has a forwardly expanding visual field [claimed i-th first point of view]”),
the processing circuitry is further configured to:
;
estimate a gaze direction of the remote operator (Otani: Para. [0088] & Fig. 13 disclose the eye tracking device 41 acquires the line-of-sight position P1 of the operator (step S4).); and
display the n (Otani: Para. [0035] discloses monitor 21 can simultaneously display, in a plurality of divided screen areas, a plurality of images acquired by the plurality of cameras 11 to 15.) such that a priority (Otani: Fig. 13 & Paras. [0091]-[0092] disclose determining a different viewpoint camera that acquires a different image to be displayed together with the enlarged image on the monitor 21 (steps S7-S8).).
However, Otani does not explicitly disclose “a i-th second point of view is defined by a combination of a predicted position and a predicted viewing direction of the i-th camera at a second timing later than the first timing,” “a homography process converts the i-th first image viewed from the i-th first point of view into a i-th second image viewed from the i-th second point of view,” and “acquire the i-th (i = 1 to n) second image by applying the homography process to the i-th first image to acquire n second images”.
Further, Kelly is in the same field of endeavor and teaches a i-th second point of view is defined by a combination of a predicted position and a predicted viewing direction of the i-th camera at a second timing later than the first timing (Kelly: Para. [0107] discloses “determining a first predicted state of the apparatus at time T [claimed second timing later than the first timing], wherein T is current time plus additional time representative of latency” and Para. [0123] discloses “producing the second virtualized view from the second viewpoint [claimed i-th second point of view]”),
a homography process converts the i-th first image viewed from the i-th first point of view into a i-th second image viewed from the i-th second point of view (Kelly: Para. [0069] discloses “An example is a “homography” operation [claimed homography process] which produces a synthetic image [claimed i-th second image] of a planar scene from one perspective given an image produced from another perspective [claimed i-th first point of view]”),
acquire the i-th (i = 1 to n) second image by applying the homography process to the i-th first image to acquire n second images (Kelly: Para. [0076] discloses “The renderer 330 accesses the rendering database 320, receives data from the rendering database 320, and produces synthetic views. Any number of synthetic views may be generated simultaneously [claimed acquire n second images] from the same rendering database 320.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Otani and Kelly before him or her, to modify the remote operation system of Otani to include the predicted position viewing direction homography process feature as described in Kelly. The motivation for doing so would have been to improve situational awareness, safety, and performance when commanding, driving, or navigating by providing a configuration that permits knowledge of the shape of the terrain ahead.
As per claim 6, Otani-Kelly disclose the remote operator terminal according to claim 1, wherein
the processing circuitry is further configured to estimate the gaze direction of the remote operator based on a direction of a line of sight of the remote operator or a steering angle of a steering operation performed by the remote operator (Otani: Para. [0043] discloses eye tracking device 41 is configured to track the movement of the operator's line of sight in a real-time manner and consecutively measure [estimate] the line-of-sight position [gaze direction] of the operator. Line-of-sight information on the line-of-sight position of the operator, the line-of-sight information being acquired by the eye tracking device 41, is input to the controller 50.).
As per claim 7, Otani-Kelly disclose the remote operator terminal according to claim 1, wherein
the homography process includes:
estimating a direction and an amount of movement of the moving body in a period from the first timing to the second timing (Kelly: Paras. [0089], [0243]-[0244] disclose the Vehicle Motion Prediction module 570, which predicts the motion of the apparatus over the terrain based on motion commands and terrain shape. The prediction must look ahead in time long enough to compensate for the actual round-trip communications latency using a forward time simulation that projects the current vehicle pose forward in time by taking small time steps.);
calculating a difference between the first point of view and the second point of view based on the direction and the amount of movement of the moving body (Kelly: Paras. [0107], [0119]-[0120] disclose the first viewpoint corresponds to the first predicted state at time T, and the second viewpoint corresponds to the second predicted state at time (T+delta T). The difference between these viewpoints is based on the predicted movement of the apparatus during the time interval (delta T), as determined by the motion prediction module.); and
converting the first image viewed from the first point of view into the second image viewed from the second point of view based on the difference between the first point of view and the second point of view (Kelly: Paras. [0069], [0169], [0183] disclose the video generation module 330 and renderer 330, which use the predicted vehicle trajectory (which defines the viewpoints) and the appearance model (encoded data) to create synthetic video.).
As per claim 8, Otani-Kelly disclose the remote operator terminal according to claim 1, wherein
the processing circuitry is further configured to:
set at least a part of a communication delay time between the moving body (12) and the remote operator terminal (14) as a delay compensation time (Kelly: Paras. [0080]-[0081], [0087]-[0088] disclose the latency predictor 410 calculates [sets] the latency in the communications between the remote apparatus 12 and the control agent 14, and vice versa.); and
set the second timing (T+delta T) to be later than the first timing (T) by the delay compensation time (Kelly: Paras. [0107], [0119]-[0120], [0286], [0289] disclose the first timing (T) is defined as current time plus additional time representative of latency for a control signal to be received and implemented by the apparatus and the delay compensation time (delta T) is a difference in a time between displaying the first virtualized view and a second virtualized view and can be set based on the refresh rate of the virtualized view. The second timing (T+delta T) is set later than the first timing (T) by the delay compensation time (delta T) because the system determines a second predicted state of the apparatus at time (T+delta T).).
As per claim 9, the claim(s) recites analogous limitations to claim(s) 1 above, and is/are therefore rejected on the same premise.
Claims 2, 4 are rejected under 35 U.S.C. 103 as being unpatentable over Otani in view of Kelly in further view of Kai et al., hereinafter referred to as Kai (CN-108038416-A).
As per claim 2, Otani-Kelly disclose the remote operator terminal according to claim 1, wherein
a magnified second image (V2, P1) is the priority second image or a second image displayed adjacent to the priority (Otani: Paras. [0074], [0078] & Figs. 8-9 disclose a magnified image (V2, P1) is a second image displayed adjacent to the priority first image (V3, P1).),
a magnified first image is a first image (V2, P1)
the processing circuitry is further configured to adjust the plurality of first images (Otani: Paras. [0074]-[0079] & Figs. 8-9 disclose a magnified image (V2, P1) larger than other first images V3.).
However, Otani-Kelly do not explicitly disclose “… a magnified first image is a first image before the homography process that serves as a source of the magnified second image in the homography process, and the processing circuitry is further configured to adjust the plurality of first images before the homography process …”.
Furthermore, Kai is in the same field of endeavor and teaches a magnified first image (the first region of interest is scaled to 480×400 pixels) is a first image before the homography process that serves as a source of the magnified second image in the homography process, and the processing circuitry is further configured to adjust the plurality of first images before the homography process (Kai: Paras. [0175]-[0176] disclose the first region of interest is first scaled to 480×400 pixels (pre-processed), then an inverse perspective transformation [homography process] is performed on the pre-processed region of interest. Therefore, adjusts images before the homography process.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Otani-Kelly and Kai before him or her, to modify the remote operation system of Otani-Kelly to include the magnifying adjusting image before the homography process feature as described in Kai. The motivation for doing so would have been to improve calculation efficiency in real-time applications by providing a configuration that optimizes transformation algorithms for image processing.
As per claim 4, Otani discloses a remote operator terminal used by a remote operator for a remote operation of a moving body (Otani: Abstract.),
wherein the remote operator terminal comprising processing circuitry configured to acquire n first images respectively captured at a first timing by n cameras mounted on the moving body (Otani: Para. [0032] discloses “a plurality of cameras 11 to 15 [claimed n cameras]” and Para. [0034] discloses “Information on images [claimed n first images] acquired by the plurality of cameras 11 to 15”.), n being an integer equal to or greater than 2 (Otani: Para. [0034] discloses “The plurality of different viewpoint cameras 12 to 15 have their respective visual fields different from the visual field of the main camera 11 [claimed n being an integer equal to or greater than 2]”.),
wherein a i-th (i = 1 to n) first image of the n first images is captured at the first timing by a i-th camera of the n cameras (Otani: Para. [0033] discloses “The main camera 11 [claimed i-th camera] acquires a main image [claimed i-th first image]”.),
a i-th first point of view is defined by a combination of a position and a viewing direction of the i-th camera at the first timing (Otani: Paras. [0033]-[0034] disclose “The main camera 11 is set, for example, at a position corresponding in height to the eyes … and has a forwardly expanding visual field [claimed i-th first point of view]”),
the processing circuitry is further configured to:
display the plurality of n second images side-by-side on a display of the remote operator terminal (Otani: Para. [0035] discloses “simultaneously display, in a plurality of divided screen areas [claimed side-by-side], a plurality of images”.),
a priority second image is one corresponding to a gaze direction of the remote operator among the plurality of n second images (Otani: Para. [0073] discloses “the part being related to the line-of-sight position [claimed priority second image corresponding to a gaze direction]”.),
a magnified second image is the priority second image or a second image displayed adjacent to the priority second image (Otani: Para. [0074] discloses “generates an enlarged image V2 [claimed magnified second image] enlarged such that a part of the work image V1 is located in the center … corresponding to a line-of-sight position P1 [claimed priority second image]”),
a magnified first image is a first image (V2, P1)
the processing circuitry is further configured to adjust the plurality of n first images(Otani: Paras. [0074]-[0079] & Figs. 8-9 disclose a magnified image (V2, P1) larger than other first images V3.).
However, Otani does not explicitly disclose “… a i-th second point of view is defined by a combination of a predicted position and a predicted viewing direction of the i-th camera at a second timing later than the first timing, a homography process converts the i-th first image viewed from the i-th first point of view into a i-th second image viewed from the i-th second point of view, … acquire the i-th (i = 1 to n) second image by applying the homography process to the i-th first image to acquire n second images; … a magnified first image is a first image before the homography process that serves as a source of the magnified second image in the homography process, and the processing circuitry is further configured to adjust the plurality of n first images before the homography process …”.
Further, Kelly is in the same field of endeavor and teaches a i-th second point of view is defined by a combination of a predicted position (predicted position state) and a predicted viewing direction (predicted orientation state) of the i-th camera (24) at a second timing later than the first timing (Kelly: Paras. [0016], [0105], [0117], [0119]; Figs. 1, 4-5, 8A-8B disclose a second viewpoint [claimed i-th second point of view] is defined by the state of apparatus 12 corresponding to the camera 24 [claimed i-th camera] at a second timing later T+delta T than the first timing T.),
a homography process converts the i-th first image viewed from the i-th first point of view (first viewpoint/perspective/viewframe) into a i-th second image (synthetic/virtualized image) viewed from the i-th second point of view (Kelly: Paras. [0066], [0069], [0106], [0123] & Figs. 4-5 disclose performing a homography operation of the given image to generate a synthetic second image or second virtualized view viewed from the second viewpoint.),
acquire the i-th (i = 1 to n) second image by applying the homography process to the i-th first image to acquire n second images (Kelly: Paras. [0069], [0079]-[0089] & Figs. 4-5 disclose performing a homography operation on incoming image data [first images] to generate a series of future-projected views [second images] that help the operator manage system latency.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Otani and Kelly before him or her, to modify the remote operation system of Otani to include the predicted position viewing direction homography process feature as described in Kelly. The motivation for doing so would have been to improve situational awareness, safety, and performance when commanding, driving, or navigating by providing a configuration that permits knowledge of the shape of the terrain ahead.
However, Otani-Kelly do not explicitly disclose “… a magnified first image is a first image before the homography process that serves as a source of the magnified second image in the homography process, and the processing circuitry is further configured to adjust the plurality of n first images before the homography process …”.
Furthermore, Kai is in the same field of endeavor and teaches a magnified first image is a first image before the homography process that serves as a source of the magnified second image in the homography process, and the processing circuitry is further configured to adjust the plurality of n first images before the homography process (Kai: Paras. [0175]-[0176] disclose the first region of interest is first scaled to 480×400 pixels (pre-processed), then an inverse perspective transformation [homography process] is performed on the pre-processed region of interest. Therefore, adjusts images before the homography process.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, and having the teachings of Otani-Kelly and Kai before him or her, to modify the remote operation system of Otani-Kelly to include the magnifying adjusting image before the homography process feature as described in Kai. The motivation for doing so would have been to improve calculation efficiency in real-time applications by providing a configuration that optimizes transformation algorithms for image processing.
Allowable Subject Matter
Claims 3 & 5 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim provided that the rejections under 35 U.S.C. 112(a) are overcome.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PEET DHILLON whose telephone number is (571)270-5647. The examiner can normally be reached M-F: 5am-1:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sath V. Perungavoor can be reached at 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PEET DHILLON/Primary Examiner
Art Unit: 2488
Date: 05-25-2026
Prosecution Insights