DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over NISHIKAWA et al. (US 20200242813 A1), referred herein as NISHIKAWA in view of Dougherty et al. (US 20200312021 A1), referred herein as Dougherty.
Regarding Claims 1, 8 and 9, NISHIKAWA in view of Dougherty teaches an information display system, method and a non-transitory computer-readable medium comprising:
at least one memory storing processing instructions; and at least one processor configured to execute the processing instructions to (NISHIKAWA Abst: A display control method comprising: detecting a real object in real scenery; FIG. 1; [0195] The processing circuit may be a processing circuit 103 as dedicated hardware, and may be a processor 102 which executes programs stored in a memory 101):
acquire position information of a display device displaying a real space image, an imaging device capturing an image of a predetermined target object, and the target object within a captured image captured by the imaging device (NISHIKAWA FIG. 1.4: display device; [0052] In the case shown in FIG. 2, the virtual object is a navigation arrow. FIG. 2 shows a situation where a vehicle that is an actual object (hereinafter, referred to as a real object) is placed between a current position of a user and a depth-direction position of the virtual object visually recognized by the user; [0057] The external information acquisition unit 10 generates the external information indicating a position, a size, etc. of the real object existing in the real scenery, by analyzing, for example, image data of the real scenery acquired from the camera 1);
NISHIKAWA disclosed user’s viewing position, but does not explicitly disclose position information of a display device. However, Dougherty teaches
acquire position information of a display device (Dougherty [0014] the client device 110 may comprise a display module (not shown) to display information (e.g., in the form of user interfaces). In further embodiments, the client device 110 may comprise one or more of touch screens, accelerometers, gyroscopes, cameras, microphones, Global Positioning System (GPS) devices, Inertial Motion Unit (IMU), and so forth; [0057] the display is based on the computing device's position in the space so that the user can view the 3D model being created as he moves the device and/or walks around the space).
NISHIKAWA in view of Dougherty further teaches
make a position of the target object within the captured image correspond to a position of the display device based on the position information (NISHIKAWA [0082] On the basis of the image data acquired from the camera 1 or the sensor data acquired from the sensor 2, the external information acquisition unit 10 detects a real object existing in the real scenery to thereby acquire the external information indicating a position, a size, etc. of the real object (Step ST1). The external information acquisition unit 10 outputs the external information to the control unit 30);
generate a virtual image of the target object based on the position information of the target object (NISHIKAWA [0052] In the case shown in FIG. 2, the virtual object is a navigation arrow. FIG. 2 shows a situation where a vehicle that is an actual object (hereinafter, referred to as a real object) is placed between a current position of a user and a depth-direction position of the virtual object visually recognized by the user); and
control the display device to display the virtual image of the target object made to correspond to the position of the display device so as to be superimposed on the real space image (NISHIKAWA [0084] The control unit 30 performs virtual-object generation processing, and outputs the image information of the thus-generated virtual object and the superimposing-position information of that virtual object to the display device 4 (Step ST3); [0101] When the control unit 30 determines that the number of changes does not reach the limit number (Step ST23; NO), the control unit 30 changes the display form of the virtual object and generates the superimposing-position information of the virtual object after the change of the display form, and then outputs the superimposing-position information of that virtual object to the to-be-hidden region acquisition unit 40 (Step ST24). When the processing in Step ST24 is completed, the flow returns again to the processing in Step ST12).
Dougherty discloses systems and methods for receiving image data via a camera of a computing device, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified NISHIKAWA to incorporate the teachings of Dougherty, and apply the position information of computing device with a display into the display device that superimposes a virtual object on real scenery.
Doing so would provide a verified inspection report identifying how a space is laid out, where various amenities are located.
Regarding Claim 2, NISHIKAWA in view of Dougherty teaches the information display system according to claim 1, and further teaches wherein the at least one processor is configured to execute the processing instructions to generate the virtual image by compressing the position information of the target object (NISHIKAWA [0070] On the basis of the external information acquired from the external information acquisition unit 10 and the superimposing-position information of the virtual object acquired from the control unit 30, the to-be-hidden region acquisition unit 40 acquires a positional relationship and a depth relationship between the superimposing position of the virtual object and the real object).
Regarding Claim 3, NISHIKAWA in view of Dougherty teaches the information display system according to claim 2, and further teaches wherein the at least one processor is configured to execute the processing instructions to generate the virtual image by simplifying a shape of the target object (NISHIKAWA [0068] When the user uses the function of highlighting a nearby vehicle or a nearby pedestrian, the image information of the virtual object indicates, for example, each frame shape as shown in FIG. 4).
Regarding Claim 4, NISHIKAWA in view of Dougherty teaches the information display system according to claim 2, and further teaches wherein the at least one processor is configured to execute the processing instructions to generate the virtual image only for the target object satisfying a previously set criterion within the captured image (NISHIKAWA [0094] The recognizability determination unit 50 determines whether or not the recognizability of the virtual object after the hiding processing is equal to or greater than the predetermined threshold value (Step ST19); [0098] The recognizability determination unit 50, when it is determined that the recognizability of the virtual object after the hiding processing is equal to or greater than the predetermined threshold value (Step ST19; YES), outputs information indicating that the recognizability is equal to or greater than the threshold value to the control unit 30 (Step ST20)).
Regarding Claim 5, NISHIKAWA in view of Dougherty teaches the information display system according to claim 2, and further teaches wherein:
the captured image captured by the imaging device is point cloud data including three-dimensional coordinates of the target object (Dougherty [0077] the computing device stores the detected object with the dimensions of the detected object (e.g., the dimensions comprising X, Y, and Z coordinate indicating the position of the at least one object in 3D space, pose indicating the position and orientation of the computing device when capturing the image frame comprising the object); [0059] As the user walks around and/or moves the computing device to scan the space, the computing device displays what has been scanned and indicates what has not been scanned, regardless of the order or direction in which the user is scanning the space. The display of the 3D reconstruction of the image data on the GUI displayed on the computing device as image data is received and 3D reconstruction of the image data is generated, providing feedback to the user so that he knows which areas he has already scanned and which areas he still needs to scan); and
the at least one processor is configured to execute the processing instructions to generate the virtual image by converting the point cloud data of the target object into a mesh, a polygon, or a bounding box (Dougherty [0056] The 3D reconstruction can be generated using any 3D reconstruction technology (e.g., augmented stereo reconstruction technology, mesh reconstruction technology, photogrammetry, neural net-based depth estimation algorithms, etc.). Generating the 3D reconstruction comprises generating a 3D mesh of the space and textures of the space (e.g., images mapped to the 3D mesh). In one example, the 3D reconstruction is generated in real time as the user is scanning the space). Point cloud data are generated from scanning.
Regarding Claim 6, NISHIKAWA in view of Dougherty teaches the information display system according to claim 2, and further teaches wherein:
the captured image captured by the imaging device is point cloud data containing three-dimensional coordinates of the target object (Dougherty [0077] the computing device stores the detected object with the dimensions of the detected object (e.g., the dimensions comprising X, Y, and Z coordinate indicating the position of the at least one object in 3D space, pose indicating the position and orientation of the computing device when capturing the image frame comprising the object); [0059] As the user walks around and/or moves the computing device to scan the space, the computing device displays what has been scanned and indicates what has not been scanned, regardless of the order or direction in which the user is scanning the space. The display of the 3D reconstruction of the image data on the GUI displayed on the computing device as image data is received and 3D reconstruction of the image data is generated, providing feedback to the user so that he knows which areas he has already scanned and which areas he still needs to scan); and
the at least one processor is configured to execute the processing instructions to measure a size of the target object based on the position information of the target object, and generate the virtual image including a measurement value (Dougherty [0071] if the area or size of the bounding box (e.g., size in 3D space) is very small (e.g., based on a number of pixels or other measure), the image may be discarded because the area may be too small to actually show such an object; NISHIKAWA [0056] The external information acquisition unit 10 generates external information indicating a position, a size, etc. of the real object existing in real scenery, and outputs the generated external information to the control unit 30 and the to-be-hidden region acquisition unit 40).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over NISHIKAWA et al. (US 20200242813 A1), referred herein as NISHIKAWA in view of Dougherty et al. (US 20200312021 A1), referred herein as Dougherty and Korkalo et al. (US 20190120963 A1), referred herein as Korkalo.
Regarding Claim 7, NISHIKAWA in view of Dougherty teaches the information display system according to claim 1. However, Korkalo teaches wherein the at least one processor is configured to execute the processing instructions to acquire, from the display device, identification information of the imaging device acquired by the display device together with the position information of the display device, and acquire position information of the imaging device associated in advance with the identification information of the imaging device (Korkalo [0020] Each camera of the system may have an unique identification code; [0029] The camera is configured to capture images or depth frames and detect a large horizontal surface from the depth frames from the depth frames. If such a surface is found, it is determined in step 310 to be the floor plane. [0030] In step 312 height of the camera 104 relative to the floor levels calculated. [0031] In step 314 the orientation (rotation matrix) 300 of the camera is determined. The rotation matrix and the camera height may be called local extrinsic parameters of the camera).
Korkalo discloses an apparatus for controlling a plurality of imaging sensor nodes producing 3D structure of a scene which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified NISHIKAWA to incorporate the teachings of Korkalo, and apply the position information and unique ID of a camera into the display device that superimposes a virtual object on real scenery.
Doing so would be able to keep track of the objects in a reliable manner.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samantha (Yuehan) Wang whose telephone number is (571)270-5011. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Samantha (YUEHAN) WANG/
Primary Examiner
Art Unit 2617
Prosecution Insights