CTNF 18/857,508 CTNF 101398 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Priority Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. Information Disclosure Statement The information disclosure statement (IDS) submitted on 10/17/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Claim Rejections - 35 USC § 102 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-12-aia AIA (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention. 07-15-aia AIA Claim(s) 1-3, and 11-12 is/are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Shigeta (WO 2021002116) . Regarding claim 1, Shigeta teaches an information processing device (“information processing device (10a)” – Abstract) comprising a control unit (“The control unit 22 controls movement of the mobile robot 20a”- Par 19, Line 1) configured to receive, at a first frequency, “This is because the position information of the mobile robot 20a is communicated at a higher frequency than the image as described above . Since the position P (ta) of the mobile robot 20a at the time ta is information that is not actually transmitted, the position P (ta) of the mobile robot 20a is obtained by interpolation using a plurality of positions P (tb) of the mobile robot 20a transmitted at a high frequency.” – Par 71, Lines 3-6 NOTE: Shigeta teaches that position information, which can be interpreted as motion information, is communicated at a higher frequency than the first captured image from the image acquisition unit. This implies that receiving the first image occurs at a first frequency. a first image generated by an imaging unit mounted on a mobile body “receives moving body information including an image (Ia) (first image) captured by a camera (26) (image capturing unit) mounted on a moving robot (20a) (moving body)” – Abstract and first time information at which the first image is generated, “The mobile body information reception unit 70 receives mobile body information including an image Ia (first image) captured by a camera 26 (imaging unit) mounted on the mobile robot 20a and position information indicating a position of the mobile robot 20a (mobile body) at a time ta at which the image Ia is captured.” – Par 48, Lines 1-2 NOTE: Shigeta teaches capturing a first image at a time ta which implies that time information is known when the image is captured. The time information is captured with the timer of the CPU 52, see par 36. receive, at a second frequency higher than the first frequency, “This is because the position information of the mobile robot 20a is communicated at a higher frequency than the image as described above . Since the position P (ta) of the mobile robot 20a at the time ta is information that is not actually transmitted, the position P (ta) of the mobile robot 20a is obtained by interpolation using a plurality of positions P (tb) of the mobile robot 20a transmitted at a high frequency.” – Par 71, Lines 3-6 NOTE: Shigeta does not explicitly teach that obtaining motion information is received at a higher frequency as the frequency of receiving the first image. However, Shigeta teaches that position information, which can be interpreted as motion information, is communicated at a higher frequency than the first captured image from the image acquisition unit. motion information regarding a motion of the mobile body generated by a sensor unit mounted on the mobile body and second time information at which the motion information is generated, “Further, the communication interface 60 acquires sensor information obtained from various sensors 29 included in the mobile robot 20a via the sensor interface 66. The various sensors 29 are a gyro sensor that measures a moving state such as a moving direction and a moving amount of the moving robot 20a, an acceleration sensor, a wheel speed sensor, a GPS (Global Positioning System) receiver, and the like.” – Par 34, Lines 1-4 NOTE: Sensors on the robot measure various motion information such as distance traveled and acceleration and velocity. It would be obvious to one of ordinary skill in the art to also use the timer of the CPU 52 that record the time of the first captured image as disclosed to record the time for when the motion information is generated, see par 36. and generate a second image based on the first image and the motion information . “Furthermore, an image generating unit (73a) generates, from the image (Ia), an image (Ib) (second image) corresponding to the movement of the moving robot (20a) indicated by the movement control information, on the basis of the movement control information received by the moving body information receiving unit (70).” – Abstract NOTE: A second image is generated based of the first captured image and the motion information recorded by the mounted sensors on the robot. To differentiate between the images, Shigeta labels the first captured image as Ia and the second generated image as Ib. Regarding claim 11, the claim recites similar limitations to claim 1. Therefore, information processing method claim 11 corresponds to the information processing device disclosed in claim 1 and is rejected for the same reasons as used above. Regarding claim 12, the claim recites similar limitations to claim 1. Therefore, information processing system claim 12 corresponds to the information processing device disclosed in claim 1 and is rejected for the same reasons as used above. Regarding claim 2, Shigeta teaches the information processing device of claim 1. Shigeta further teaches wherein the second image is viewed by an operator who remotely operates the mobile body (“Hereinafter, an information processing system that presents an image captured by a camera installed in a mobile robot to a remote operator who operates the mobile robot from a remote location (hereinafter, referred to as an operator) will be described” – Par 12). Regarding claim 3, Shigeta teaches the information processing device of claim 2. Shigeta further teaches wherein the control unit estimates mobile body position information regarding a position and an attitude of the mobile body “The self-position calculated by the mobile robot 20a may include information on the orientation of the mobile robot 20a.” - Par 35, Lines 3-4 NOTE: Shigeta discloses a self-position estimation unit on the mobile robot which estimates the current position and time of the mobile robot, see par 59-62. The self-position estimation unit may also include orientation of the robot in the space which can be interpreted as the attitude information of the mobile body. at a time at which the operator views the second image based on the motion information, “As a result, the information processing device 10a can accurately estimate the current position P (t) of the mobile robot 20a in consideration of the operation instruction given to the mobile robot 20a by the operator 50. Further, in the information processing apparatus 10a, the display control unit 74 displays the image Ib (second image) on the HMD 16. This allows the operator 50 to observe an image with a sense of reality .” – Par 123-125 NOTE: The second image, which considers the estimation of position after a given operation instruction, is displayed to the operator who is wearing a head mounted display. The second image corresponds to movement of the mobile body from the first image, see par 8. and generates the second image based on the first image and the mobile body position information. “In the information processing apparatus 10a, the image generation unit 73a generates an image Ib (second image) corresponding to the current position P (t) of the mobile robot 20a (moving body) estimated by the current position estimation unit 72 from the image Ia (first image).” – Par 116 Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shigeta in view of Sato et al (JP6961888), hereinafter Sato Regarding claim 5, Shigeta teaches the information processing device of claim 1. Shigeta does not teach wherein the control unit generates each of a plurality of the second images generated from one first image so that a resolution of each of the plurality of second images falls within a predetermined range. However, Sato teaches wherein the control unit generates each of a plurality of the second images generated from one first image so that a resolution of each of the plurality of second images falls within a predetermined range. “The circuit produces a plurality of images that have a higher resolution than the plurality of second resolution images and are output among the plurality of first resolution images in the moving image based on the plurality of first resolution images and motions.”- Abstract NOTE: Sato discloses an imaging apparatus that receives first images of a first resolution image and second images of a second resolution lower than the first images. A circuit is used to generate new images that are based on the first image but at a higher resolution than the second images’ resolution. It can be interpreted that the range set for these newly generated images is equal to or higher than the resolution of the second images received by the imaging apparatus. After the combination, the generation of new images based on the first received images set at predetermined range taught by Sato can be used by the information processing device as taught by Shigeta. This would allow the image captured by the mobile robot of Shigeta to then be used as the basis for the generation of new images at a predetermined resolution range. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Shigeta by incorporating the teachings of Shigeta to generate new images based on the first image captured so that the resolution of the newly generated images fall within predetermined range. This would have the predicted result in maintaining acceptable image visibility for the operator . 07-21-aia AIA Claim (s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shigeta and Usami (US 20040212686 A1)), hereinafter Usami . Regarding claim 6, Shigeta teaches the information processing device of claim 1. Shigeta does not teach wherein the control unit corrects blurring of the first image caused by a motion of the mobile body in a vertical direction, and applies processing corresponding to vibration caused by the motion of the mobile body in the vertical direction to the second image. However, Usami teaches wherein the control unit corrects blurring of the first image caused by a motion of the mobile body in a vertical direction, “Still further, in the vehicle-mounted camera apparatus according to the present invention, it is preferred that the image motion blur corrector determines the amount and direction of the motion blur in the image displayed on the screen that corresponds to the image captured by said camera based on the vibrations detected by the vibration detector” – Par 16 NOTE: Usami discloses image motion blur corrector that determines the amount and direction of the motion blur and then corrects the image before it is displayed. After the combination, the concept of an image blur corrector that corrects blur based on the amount and direction of the motion blur caused by vibration as taught by Usami can be used by the imaging device on the mobile robot as taught by Shigeta to correct the first images motion blur that may occur from moving the mobile body. This would allow a clear visual of what the mobile robot sees for the operator to continue transmitting control operations regarding movement. and applies processing corresponding to vibration caused by the motion of the mobile body in the vertical direction to the second image. “Still further, in the vehicle-mounted camera apparatus according to the present invention, it is preferred that the image motion blur corrector determines the amount and direction of the motion blur in the image displayed on the screen that corresponds to the image captured by said camera based on the vibrations detected by the vibration detector” – Par 16 NOTE: After the combination, the image motion blur corrector as taught by Usami can apply blur correction processing to the second image generated based on the first image captured and obtained motion information of the mobile robot as taught by Shigeta. This would allow further blur correction processing on the second image, similarly done to the first captured image, to clear the visuals of the mobile robot’s captured images for the operator. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Shigeta by incorporating the teachings of Usami to correct the vertical direction motion blur of the first image and apply the same blur correction image processing to the second image. This would have the predicted result of removing inaccuracies of the captured image of the mobile robot’s view which will give the remote operator clear visual of the robot’s surroundings . 07-21-aia AIA Claim (s) 7 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shigeta and Wang et al (US 20120197464 A1) hereinafter Wang . Regarding claim 7, Shigeta teaches the information processing device of claim 2. Shigeta further teaches an output unit configured to display the second image, “The display control unit 74 displays the image Ib on a display unit 90 (a display panel such as an LCD or an OLED) included in the HMD 16 via an image output interface such as an HDMI” - Par 54 NOTE: The HMD device is used to display the second image generated based on the first captured image and motion information. Shigeta does not teach wherein the control unit, when the second image is displayed on the output unit, performs control so that the second image is displayed on the output unit in a display mode in which the operator is able to visually recognize that the second image is displayed. However, Wang teaches wherein the control unit, when the second image is displayed on the output unit, performs control so that the second image is displayed on the output unit in a display mode in which the operator is able to visually recognize that the second image is displayed “The user interface 605 includes a remote video feed window 610 displaying a remote view 612, such as a video feed of a patient 614. The video feed may be generated by one of the cameras 320, 450 on the robot 100. The user interface 605 may display a plan view map window 620 having a map 622 of the local area in which the robot 100 is operating. ” – Par 215 and Fig 6C NOTE: Fig 6C shows user interface elements as seen by the remote operator. Some of these elements include camera feeds and images. This corresponds functionally to a display mode on the output unit which is viewed by the remote operator. After the combination, the UI elements displayed to the remote operator as taught by Wang can be used to display the second image generated by the first image and motion information as taught by Shigeta. One of ordinary skill in the art could have the second image be displayed as an UI element that would be visually recognizable as the second image. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Shigeta by incorporating the teachings of Wang to have an output unit display the second image and perform control so that the second image is in a display mode that will allow the operator to visually recognize the second image. One would be motivated to make this combination so that the remote operator will be able to clearly identify the generated second image which shows the prediction of the mobile robot’s position based on the first captured image and the motion information. Regarding claim 10, Shigeta teaches the information processing device of claim 1. Shigeta does not teach wherein the control unit performs control so that an image corresponding to a dashboard of the mobile body is superimposed on the second image and displayed. However, Wang teaches wherein the control unit performs control so that an image corresponding to a dashboard of the mobile body is superimposed on the second image and displayed. “ FIG. 19 illustrates an exemplary remote view 612 where the telepresence software application 601 overlays a screen indicator 1910 on the remote video feed received from the robot 100.” – Par 329, Lines 1-4 NOTE: Wang discloses a remote controlled mobile robot that allows the operator to see the current field of view alongside augmented overlays and UI elements. This can be interpreted as a dashboard of the mobile body that is superimposed on an image, see par 216. After the combination, the concept of displaying augmented overlays and Ui elements displayed alongside the robot’s view as taught by Wang can be used by Shigeta to have some dashboard display of various overlays and UI elements to the second image. One motivation for doing this is to have information regarding motion be overlayed on top of the second image for the operator to view. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Shigeta by incorporating the teachings of Wang to superimpose and display a dashboard of the mobile body on the second image. This would have the predicted result of providing the operator with various information regarding position and motion that may assist in deciding the next control information to transmit . 07-21-aia AIA Claim (s) 8 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shigeta and Eguchi (US 12241232 B2), hereinafter Eguchi . Regarding claim 8, Shigeta teaches the information processing device of claim 2. Shigeta does not teach an output unit configured to display the second image, wherein the control unit, when the sensor unit detects an object located within a predetermined range from the mobile body, performs control so that information notifying the operator that the object is detected is displayed on the output unit. However, Eguchi teaches an output unit configured to display the second image, wherein the control unit, when the sensor unit detects an object located within a predetermined range from the mobile body, performs control so that information notifying the operator that the object is detected is displayed on the output unit “In addition, when a plurality of obstacles is detected within a predetermined range around the work vehicle 100, the control device 143 according to the first embodiment causes to display a display image in which markers are disposed at positions related to a predetermined number of obstacles having the highest detection accuracy within the range.” – Col 11, Lines 60-65 NOTE: Eguchi teaches a camera and sensor unit that detects an object such as people near a vehicle and provides display markers on the display image if the person is within a predetermined range to notify the operator. After the combination, the display image showing the marked object within a predetermined range as taught by Eguchi can be displayed to the mobile robot’s operator via the head-mounted display device as taught by Shigeta. This would allow the remote operator to quickly identify objects in the generated second image that may need to be avoided. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Shigeta by incorporating the teachings of Eguchi to display the second image and notify the remote operator of objects within the predetermined range from the mobile body. One would be motivated to make this combination to prevent the remote operator from hitting other objects when controlling the mobile body. Regarding claim 9, Shigeta in view of Eguchi teaches the information processing device of claim 8. Shigeta does not teach wherein the control unit performs control so that a region where the object appeared is visually emphasized and displayed on the output unit. However, Eguchi further teaches wherein the control unit performs control so that a region where the object appeared is visually emphasized and displayed on the output unit “Incidentally, the marker disposition unit 215 according to another embodiment may dispose a marker within a predetermined range from the detection position. Exemplary examples of the shape of the marker include a circular shape, an elliptical shape, a regular polygon shape, a polygon shape, etc. Incidentally, the marker is one example of an obstacle-identifying display. In another embodiment, for example, the obstacle-identifying display may be any obstacle-identifying display such as a display of a text indicating a person, a change in the color of a person, and a change in the shape of a person.” – Col 6, Lines 60-65 and Col 7, Lines 1-3 NOTE: Eguchi discloses that a marker is used to emphasize the position of the object. The marker may be modified in various ways to visually stand out by changing shape or color. The display may also show text that indicates the object. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to modify Shigeta by incorporating the teachings of Eguchi to visually emphasize the region of the object and display it to the output unit. One would be motivated to make this combination to allow the remote operator of the mobile body to clearly distinguish the object from the background. This may assist the remote operator in object avoidance when moving the mobile body . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim 4 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims Regarding claim 4, Shigeta teaches the method of claim 3. Shigeta further teaches wherein the control unit estimates view frustum position information regarding a position and an attitude of a view frustum at a time at which the operator views the second image based on the motion information, “When the position information received by the mobile body information reception unit 70 includes information about the orientation of the mobile robot 20a, the image generation unit 73a may use the orientation information when generating the image Ib (second image).” – Par 53, Lines 1-3 NOTE: Shigeta discloses a self-position estimation unit on the mobile robot which estimates the current position and time of the mobile robot, see par 59-62. The self-position estimation unit may also include orientation of the robot, see par 35. The camera field of view that captures the first image used as a basis for the generation for the second image inherently defines a view frustum which can be interpreted as the view of what the camera can see. This view frustum that captures the first image is used alongside corresponding motion information to generate the second image which shows an estimated position and attitude of the mobile robot’s view. Shigeta, neither singularly or in combine with other references teach “generating a region as the second image in the first image disposed in a virtual space, the region disposed in the virtual space and intersecting the view frustum disposed at the position and attitude indicated by the view frustum position information.” When used in combine with other claimed limitations. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID V. NGUYEN whose telephone number is (571)272-6111. The examiner can normally be reached M-F 7:30-5:00. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /DAVID VAN NGUYEN/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617 Application/Control Number: 18/857,508 Page 2 Art Unit: 2617 Application/Control Number: 18/857,508 Page 3 Art Unit: 2617 Application/Control Number: 18/857,508 Page 4 Art Unit: 2617 Application/Control Number: 18/857,508 Page 5 Art Unit: 2617 Application/Control Number: 18/857,508 Page 6 Art Unit: 2617 Application/Control Number: 18/857,508 Page 7 Art Unit: 2617 Application/Control Number: 18/857,508 Page 8 Art Unit: 2617 Application/Control Number: 18/857,508 Page 9 Art Unit: 2617 Application/Control Number: 18/857,508 Page 10 Art Unit: 2617 Application/Control Number: 18/857,508 Page 11 Art Unit: 2617 Application/Control Number: 18/857,508 Page 12 Art Unit: 2617 Application/Control Number: 18/857,508 Page 13 Art Unit: 2617 Application/Control Number: 18/857,508 Page 14 Art Unit: 2617 Application/Control Number: 18/857,508 Page 15 Art Unit: 2617 Application/Control Number: 18/857,508 Page 16 Art Unit: 2617
Prosecution Insights