Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/05/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-17, 19-21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-10, 12-17, 20 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US PGPUB 20160140394) in view of Katori et al. (US PGPUB 20180240220).
[Claim 1]
Lee teaches an information processing device comprising:
Circuitry (Paragraph 24) configured to:
determine a composition position of an auxiliary image in a second captured image captured after a first captured image by calculating a predicted position of a subject in the second captured image in accordance with a position and a motion of the subject in the first captured image and determining the predicted position as the composition position (Paragraph 33, figs. 2, 3a and 4a illustrate a position estimation and detection of the target object in a current video frame 202, in accordance with an embodiment. In the illustrated examples, an estimate determined with a motion model predicts the target object has motion vector 312 and will move from a position associated with bounding box 210 within frame 201 to an estimated position associated with bounding box 315 within frame 202. Paragraph 35, Referring again to FIG. 3A the object detection algorithm beginning at the position associated with bounding box 315 iterates until converging to a detected object position 521 associated bounding box 320.); and
composite the auxiliary image with the composition position in the second captured image (Paragraph 35, Referring again to FIG. 3A the object detection algorithm beginning at the position associated with bounding box 315 iterates until converging to a detected object position 521 associated bounding box 320).
Lee fails to teach measuring a display delay time from imaging to display and determining the predicted position after the display delay time.
However Katori teaches The real object detection unit 100 has a function of detecting information regarding the real object at the detection time T.sub.0. For example, the real object detection unit 100 acquires information that has been detected by a sensor that senses the real object as a sensing target. As such a sensor, a camera, a depth sensor, an infrared ray sensor, a radio wave sensor, and the like, for example, are exemplified (Paragraph 66). The depiction delay detection unit 300 detects how long the display time T.sub.2 is delayed with respect to the detection time T.sub.0. The depiction delay detection unit 300 compares a time stamp at a timing at which each of the real object detection unit 100 and the viewpoint detection unit 200 acquires information of the real space with a time stamp at a timing at which a display controller 704 outputs an image on the basis of the acquired information and detects the difference therebetween as a delay (ΔT). The depiction delay detection unit 300 provides the detected delay to each of the real object prediction unit 140, the viewpoint prediction unit 240, and the state-in-sight prediction unit 400 (Paragraph 182). Specifically, the state-in-sight prediction processing unit 402 predicts at which position and posture the real object as a target of superimposition with the virtual object (auxiliary image) is present in the field of vision at the timing (display time T.sub.2) at which the display controller 704 outputs the image (Paragraph 183).
Therefore taking the combined teachings of Lee and Katori, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have measured a display delay time from imaging to display and determining the predicted position after the display delay time in order to suppress disturbance of display of a virtual object or an auxiliary image by performing display control based on a degree of prediction accuracy related to the aforementioned time lag, and even in a case in which the display is disturbed, it is possible to reduce confusion or an unpleasant feeling given to the user.
[Claim 2]
Lee teaches wherein the composition position determiner circuitry determines the position of the subject on the second captured image as the composition position on a basis of the position of the subject in the first captured image (Paragraph 33, figs. 2, 3a and 4a, FIG. 3A and 4A illustrate a position estimation and detection of the target object in a current video frame 202, in accordance with an embodiment. In the illustrated examples, an estimate determined with a motion model predicts the target object has motion vector 312 and will move from a position associated with bounding box 210 within frame 201 to an estimated position associated with bounding box 315 within frame 202. Paragraph 35, Referring again to FIG. 3A the object detection algorithm beginning at the position associated with bounding box 315 iterates until converging to a detected object position 521 associated bounding box 320) and a difference between imaging timings of the first captured image and the second captured image (Paragraph 25, object tracking entails object detection over a time sequence, through which a temporal sequence of position coordinates associated with motion of the object across consecutive frames of image data is generated. Beyond position, other object features may also be updated as part of a state vector tracking one or more of object size, color texture, shape, etc. In “real-time” visual object tracking, an image data (video) stream is analyzed frame-by-frame concurrently with frame-by-frame generation or receipt of the stream).
[Claim 3]
Lee teaches wherein the motion of the subject is identified by using a captured image captured before the first captured image (Paragraph 33, , an estimate determined with a motion model predicts the target object has motion vector 312 and will move from a position associated with bounding box 210 within frame 201 to an estimated position associated with bounding box 315 within frame 202).
[Claim 4]
Lee teaches wherein the auxiliary image is a frame image indicating a specific position of the subject (bounding box 320, fig. 3a).
[Claim 5]
Lee teaches wherein the frame image is an image indicating a focus position (Paragraph 25, In further embodiments, at least the positional information associated with a tracked object is passed to a 3A (automatic focus, automatic exposure, automatic white balance) engine that manages further processing of the image frame(s)).
[Claim 6]
Lee teaches wherein the auxiliary image is an image indicating a position of a specific subject recognized as a result of image recognition processing on the first captured image (Paragraph 33, FIG. 3A and 4A illustrate a position estimation and detection of the target object in a current video frame 202, in accordance with an embodiment. In the illustrated examples, an estimate determined with a motion model predicts the target object has motion vector 312 and will move from a position associated with bounding box 210 within frame 201 to an estimated position associated with bounding box 315 within frame 202).
[Claim 7]
Lee teaches wherein the information processing device serves as a smartphone including an imaging unit that captures the first captured image and the second captured image (Paragraph 52).
[Claim 8]
Lee teaches wherein the first captured image and the second captured image are preview images displayed on a display unit (Paragraph 58, As further illustrated in FIG. 6, target object data may be output to storage/display/transmission pipeline 695. In one exemplary storage pipeline embodiment, target object data is written to electronic memory 620 (e.g., DDR, etc.) to supplement stored input image data. Memory 620 may be separate or a part of a main memory 610 accessible to APU 650. Alternatively, or in addition, storage/display/transmission pipeline 695 is to transmit target object data and/or input image data off video capture device 503).
[Claim 9]
This is a computer-readable medium corresponding to apparatus claim 1 and is therefore analyzed and rejected based upon apparatus claim 1.
[Claim 10]
Lee teaches an information processing device comprising:
a first processor (Paragraph 56, DSP 685 and/or applications processor (APU) 650 implements one or more of the validated model object tracking device modules depicted in FIG. 5) that performs image processing on a captured image output from a pixel array (image sensor 659, fig. 6) in which pixels each having a photoelectric conversion element are two- dimensionally arranged (Paragraph 29, The number of pixel values within one frame of image data depends on the input image resolution, which in further embodiments is a function of a local CM. Although embodiments herein are applicable to any input image resolution, in an exemplary embodiment the input image data is at least a 1920×1080 pixel (2.1 megapixel) representation of an image frame (i.e. Full HD)) and processing of determining a composition position of an auxiliary image to be composited with the captured image on a basis of the captured image (Paragraph 33, figs. 2, 3a and 4a, FIG. 3A and 4A illustrate a position estimation and detection of the target object in a current video frame 202, in accordance with an embodiment. In the illustrated examples, an estimate determined with a motion model predicts the target object has motion vector 312 and will move from a position associated with bounding box 210 within frame 201 to an estimated position associated with bounding box 315 within frame 202. Paragraph 35, Referring again to FIG. 3A the object detection algorithm beginning at the position associated with bounding box 315 iterates until converging to a detected object position 521 associated bounding box 320.); and
a second processor (Paragraph 58, As further illustrated in FIG. 6, target object data may be output to storage/display/transmission pipeline 695 from DSP 685) that performs processing of displaying, on a display, a composite image obtained by compositing the auxiliary image with the composition position in an image subjected to the image processing, wherein determining the composition position includes calculating a predicted position of a subject in a second captured image captured after a first captured image in accordance with a position and a motion of the subject in the first captured image and determining the predicted position as the composition position (Paragraph 33, figs. 2, 3a and 4a, FIG. 3A and 4A illustrate a position estimation and detection of the target object in a current video frame 202, in accordance with an embodiment. In the illustrated examples, an estimate determined with a motion model predicts the target object has motion vector 312 and will move from a position associated with bounding box 210 within frame 201 to an estimated position associated with bounding box 315 within frame 202. Paragraph 35, Referring again to FIG. 3A the object detection algorithm beginning at the position associated with bounding box 315 iterates until converging to a detected object position 521 associated bounding box 320.).
Lee fails to teach measuring a display delay time from imaging to display and determining the predicted position after the display delay time.
However Katori teaches The real object detection unit 100 has a function of detecting information regarding the real object at the detection time T.sub.0. For example, the real object detection unit 100 acquires information that has been detected by a sensor that senses the real object as a sensing target. As such a sensor, a camera, a depth sensor, an infrared ray sensor, a radio wave sensor, and the like, for example, are exemplified (Paragraph 66). The depiction delay detection unit 300 detects how long the display time T.sub.2 is delayed with respect to the detection time T.sub.0. The depiction delay detection unit 300 compares a time stamp at a timing at which each of the real object detection unit 100 and the viewpoint detection unit 200 acquires information of the real space with a time stamp at a timing at which a display controller 704 outputs an image on the basis of the acquired information and detects the difference therebetween as a delay (ΔT). The depiction delay detection unit 300 provides the detected delay to each of the real object prediction unit 140, the viewpoint prediction unit 240, and the state-in-sight prediction unit 400 (Paragraph 182). Specifically, the state-in-sight prediction processing unit 402 predicts at which position and posture the real object as a target of superimposition with the virtual object (auxiliary image) is present in the field of vision at the timing (display time T.sub.2) at which the display controller 704 outputs the image (Paragraph 183).
Therefore taking the combined teachings of Lee and Katori, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have measured a display delay time from imaging to display and determining the predicted position after the display delay time in order to suppress disturbance of display of a virtual object or an auxiliary image by performing display control based on a degree of prediction accuracy related to the aforementioned time lag, and even in a case in which the display is disturbed, it is possible to reduce confusion or an unpleasant feeling given to the user.
[Claim 12]
Lee teaches wherein the first processor executes processing of compositing the auxiliary image with the image subjected to the image processing (Paragraphs 33, 57-59).
[Claim 13]
Lee teaches wherein the auxiliary image is a frame image indicating a specific position of a subject (bounding box 320, fig. 3a).
[Claim 14]
Lee teaches wherein the frame image is an image indicating a focus position (Paragraph 25, In further embodiments, at least the positional information associated with a tracked object is passed to a 3A (automatic focus, automatic exposure, automatic white balance) engine that manages further processing of the image frame(s)).
[Claim 15]
Lee teaches wherein the auxiliary image is an image indicating a position of a specific subject recognized as a result of image recognition processing on the captured image (Paragraph 33, FIG. 3A and 4A illustrate a position estimation and detection of the target object in a current video frame 202, in accordance with an embodiment. In the illustrated examples, an estimate determined with a motion model predicts the target object has motion vector 312 and will move from a position associated with bounding box 210 within frame 201 to an estimated position associated with bounding box 315 within frame 202).
[Claim 16]
Lee teaches wherein the information processing device serves as a smartphone including the pixel array (Paragraphs 52 and 53).
[Claim 17]
This is a computer-readable medium corresponding to apparatus claim 10 and is therefore analyzed and rejected based upon apparatus claim 10.
[Claim 20]
Lee teaches wherein the circuitry is configured to complete determining the composition position of the auxiliary image before completing image processing for generating the second captured image as a preview image (In fig. 2, in the first frame, the composition position of the auxiliary image 210 has been completed before the next frame in fig. 3a).
[Claim 21]
Katori teaches wherein the auxiliary image (virtual object) is provided with accompanying data including at least one of a frame identification and a time stamp identifying a captured image used to generate the auxiliary image, and the circuitry is configured to measure the display delay time based on the accompanying data (Paragraph 182, The depiction delay detection unit 300 detects how long the display time T.sub.2 is delayed with respect to the detection time T.sub.0. The depiction delay detection unit 300 compares a time stamp at a timing at which each of the real object detection unit 100 and the viewpoint detection unit 200 acquires information of the real space with a time stamp at a timing at which a display controller 704 outputs an image on the basis of the acquired information and detects the difference therebetween as a delay (ΔT). The depiction delay detection unit 300 provides the detected delay to each of the real object prediction unit 140, the viewpoint prediction unit 240, and the state-in-sight prediction unit 400) in order to suppress disturbance of display of a virtual object or an auxiliary image by performing display control based on a degree of prediction accuracy related to the aforementioned time lag, and even in a case in which the display is disturbed, it is possible to reduce confusion or an unpleasant feeling given to the user.
Claim(s) 11 is rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US PGPUB 20160140394), Katori et al. (US PGPUB 20180240220) and in further view of Netsu (US PGPUB 20170223223).
[Claim 11]
Lee in view of Katori fails to teach wherein the first processor performs the image processing and the processing of determining the composition position in parallel. However Netsu teaches that in addition, image processes, such as color conversion, oblique motion correction, and dust removal, in addition to the image synthesis may be performed either before or after the image synthesis is performed, or may be simultaneously performed (Paragraph 64). Therefore taking the combined teachings of Lee, Katori and Netsu, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have the first processor performs the image processing and the processing of determining the composition position in parallel in order to have a fast process that saves time.
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US PGPUB 20160140394), Katori et al. (US PGPUB 20180240220) and in further view of Yamamoto et al. (JP 2018163700, Published on Sep. 3, 2018), US PGPUB 20210306586 is used for translation.
[Claim 19]
Lee in view of Katori fails to teach wherein the circuitry is configured to determine whether the subject is moving, and execute the calculation of the predicted position only when it is determined that the subject is moving. However Yamamoto based on the determination as to whether an object recognized in the frame 500a is a stationary body or moving body, it is also possible to predict the position of the object 500a in the next frame 500b, enabling restriction of the readout region to be read out next based on the predicted position. Furthermore, at this time, in a case where the recognized object is a moving body, further predicting the speed of the moving body will make it possible to restrict the readout region to be read out next with higher accuracy (Paragraph 581). If the object is stationary, then the position will not change. Therefore taking the combined teachings of Lee, Katori and Yamamoto, it would be obvious to one skilled in the art before the effective filing date of the invention to have been motivated to have determine whether the subject is moving, and execute the calculation of the predicted position only when it is determined that the subject is moving in order to read out when the subject is moving thereby saving power.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YOGESH K AGGARWAL whose telephone number is (571)272-7360. The examiner can normally be reached Monday - Friday 9:30-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sinh Tran can be reached at 5712727564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YOGESH K AGGARWAL/ Primary Examiner, Art Unit 2637
Prosecution Insights