DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Office Action is in response to the Applicants' communication filed on February 21, 2025. In virtue of this communication, claims 1-20 have been canceled; claims 1-20 are currently presented in the instant application.
Drawings
The drawings submitted on February 21, 2025. These drawings are reviewed and accepted by the examiner.
Information Disclosure Statement
The information Disclosure Statement (IDS) Form PTO-1449, filed on February 21, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosed therein was considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention does not fall within one of the four categories of invention that Congress deemed to be the appropriate subject matter of a patent (processes, machines, manufactures, or compositions of matter).
In particular, claim 20 recites “A storage medium storing a program for causing a computer to function as a control unit that performs: processing of superimposing and displaying a second image generated from point cloud data including feature point data of an object in a real space on a first image generated on a basis of outer circumferential line data of an object acquired from two-dimensional map data of the real space; and processing of updating geographic coordinate information of the point cloud data on a basis of a result of a matching operation of the second image with the first image …” which includes not only non-transitory embodiments, such as a physical disk drive, but also includes transitory embodiments, such as an electromagnetic signal or carrier wave encoded with (and thus storing) storing a program for causing a computer to function as a control unit that performs, which has been deemed not being directed to one of the statutory classes of invention. See MPEP § 2106.03(1): “Non-limiting examples of claims that are not directed to any of the statutory.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 13-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1).
Regarding claim 1. Minear discloses an information processing device comprising a control unit that performs:
processing of superimposing and displaying a second image generated from point cloud data including feature point data of an object in a real space on a first image generated on a basis of outer circumferential line data of an object acquired from two-dimensional map data of the real space (Minear, see at least par. [0021] In the present invention, a 2D image is modified and then fused with 3D point cloud data for a common scene. The process facilitates analysis of the scene and permits improved evaluation of the quality of the image registration process. The 2D image and 3D point cloud data are registered utilizing a suitable registration process. Thereafter, the 3D point cloud data is processed to identify and define a ground table. The ground table represents a contour of the ground in the scene. The ground table is then used to transform the 2D image into a virtual 3D image comprising a ground surface contour. The virtual 3D image is created by selectively modifying a Z value associated with the pixels of the 2D image so that they generally conform to the Z values defined by the ground table. In this way, the contours of the ground surface are imparted to the 2D image, thereby forming the virtual 3D image. Once this process is complete, the previously registered 3D point cloud data is overlaid on the virtual 3D image. The resulting fused 3D image offers improved visualization of the scene and permits better evaluation of the quality of the registration.); and
Minear does not disclose processing of updating geographic coordinate information of the point cloud data on a basis of a result of a matching operation of the second image with the first image. However,
Chen discloses:
processing of updating geographic coordinate information of the point cloud data on a basis of a result of a matching operation of the second image with the first image (Chen, see at least par. [0050] In certain embodiments, the collected 3D point cloud data is further processed with the collected 2D image data. In certain embodiments, the 2D color images are used to colorize the 3D point cloud. The colorization process may involve overlaying a 3D point cloud with a 2D image of the same scene. For example, 3D LIDAR point clouds are overlaid with 2D color images from the same perspectives. Based on this matching of 2D and 3D data, the 3D point cloud data may be colorized to form a 3D colored point cloud.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have processing of updating geographic coordinate information of the point cloud data on a basis of a result of a matching operation of the second image with the first image, as provided by Chen. The modification provides an improved system and method for superimposing and displaying a virtual object on a real
space in a state where a user can visually recognize the real space in front of the user, thereby to provided for accurately range sensing structures at least partially made of a transparent material. (Chen, see par. [0005]).
Regarding claim 4. Minear in view of Chen discloses the information processing device according to claim 1, and Minear in view of Chen wherein the geographic coordinate information includes at least information indicating latitude, longitude, and an azimuth angle (Chen, see at least par. The positioning system 213 is a GPS system, one or more mechanical and/or electrical sensors, one or more gyroscopes, a local positioning system, one or more direction sensors, or another system or device for providing position data, such as location data (e.g., longitude, latitude, and/or altitude) and/or direction data, of the device 201, components thereof, or an attached object, such as a vehicle.).
Regarding claim 13. Minear in view of Chen discloses the information processing device according to claim 1, and Minear in view of Chen further discloses wherein the point cloud data includes data used when position estimation is performed by a visual positioning system (VPS) (Zhang, see at least par. [0154] Graphics subsystem 1315 may perform processing of images such as still or video for display. Graphics subsystem 1315 may be a graphics processing unit (GPU) or a visual processing unit (VPU), for example. An analog or digital interface may be used to communicatively couple graphics subsystem 1315 and display 1320. For example, the interface may be any of a High-Definition Multimedia Interface, Display Port, wireless HDMI, and/or wireless HD compliant techniques. Graphics subsystem 1315 may be integrated into processor 1310 or chipset 1305. In some implementations, graphics subsystem 1315 may be a stand-alone card communicatively coupled to chipset 1305.).
Regarding claim 14. Minear in view of Chen discloses the information processing device according to claim 1, and Minear in view of Chen wherein the two-dimensional map data includes geospatial information in a vector tile format provided from an external two-dimensional map data service (Chen, see at least par. [0054] LIDAR point clouds may also undergo point cloud smoothing during processing to minimize or remove 3D data points that do not accurately represent a geographic area, or structures in a geographic area. Any point cloud smoothing technique may be used. In an embodiment, a moving least squares technique is used to minimize a weighted least square error for each 3D data point of the point cloud.).
Regarding claim 15. Minear in view of Chen discloses the information processing device according to claim 1, and Minear in view of Chen further discloses wherein the object in the real space includes at least a building, a road, or a river (Chen, see at least par. [0026] The object 104 is a vehicle, such as a car, truck, motorcycle, bicycle, Segway, or other transportation device or mechanical device used to move on the path network 108. In one embodiment, the object 104 is a robot that is controlled by a user or moves without user interaction. Alternatively, the object 104 may represent a pedestrian, such as a human being or animal that walks or moves along the path network 108. The path network 108 is a road network and/or a collection of other pathways. For example, the path network 108 is a road network made up of various roads. The roads may be used to drive vehicles on, such as local and neighborhood streets as well as highways. Also, instead of or in addition to traditional streets, the path network 108 may include bicycle roads or paths, walking paths, or other travel paths. Alternatively, the path network 108 may be an open area space with no specific paths or travel constraints. The path network 108 is in the geographic region 112, such as a city, a suburb, a state, a country, and/or other geographic region. In one embodiment, the path network 108 is a collection of pathways within a building. The pathways within the building may, for example, be hallways, rooms, lobbies, or any other pathway within a building.).
Regarding claim 16. Minear in view of Chen discloses the information processing device according to claim 1, and Minear in view of Chen further discloses wherein the control unit generates the first image and the second image from a viewpoint from an angle other than an angle at which the first image and the second image are viewed from a horizontal plane (Chen, see at least par. [0003] A two-dimensional (2D) image may be generated by, for example, a camera, from the same viewpoint as the 3D point cloud and may be used to colorize the 3D point cloud. The generated 2D image and the 3D point cloud may be overlaid and processed to form a colorized 3D point cloud having color pixels attributed to the x, y, and z coordinates for the 3D point cloud.).
Regarding claim 17. Minear in view of Chen discloses the information processing device according to claim 1, and Minear in view of Chen further discloses wherein the geographic coordinate information includes information indicating latitude, longitude, an azimuth angle, an elevation, and an elevation angle (Chen, see pars. [0035], The road segments and nodes may be associated with attributes, such as geographic coordinates, street names, address ranges, speed limits, turn restrictions at intersections, and other vehicle navigation attributes, as well as POIs, such as gasoline stations, hotels, restaurants, museums, stadiums, offices, automobile dealerships, auto repair shops, buildings, stores, etc. Alternatively, the geographic database 136 contains path segment/link and node data records or other data that represent pedestrian paths and/or building hallways and rooms in addition to or instead of the vehicle road record data. [0041] The positioning system 213 is a GPS system, one or more mechanical and/or electrical sensors, one or more gyroscopes, a local positioning system, one or more direction sensors, or another system or device for providing position data, such as location data (e.g., longitude, latitude, and/or altitude) and/or direction data, of the device 201, components thereof, or an attached object, such as a vehicle.).
Regarding claim 19. Minear in view of Chen discloses an information processing method comprising: by a processor (Minear, see FIG. 1 and par. [0027], “Processor 102”) to perform the same steps of claim 1. Therefore, claim 19 is further rejected based on the same rationale as claim 1 set forth above and incorporated herein.
Regarding claim 20. A storage medium storing a program for causing a computer to function as a control unit that performs the same steps of claim 1. Therefore, claim 20 is further rejected based on the same rationale as claim 1 set forth above and incorporated herein.
Claims 2 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1), as applied claim 1 above and further in view of Zhang et al. (US 20200388004 A1)
Regarding claim 2. Minear in view of Chen discloses the information processing device according to claim 1, but Minear in view of Chen does not discloses wherein the matching operation includes an adjustment operation of at least a position or rotation with respect to the second image performed by a user so that a point cloud drawn in the second image matches a shape of an object drawn in the first image. However,
Zhang discloses:
wherein the matching operation includes an adjustment operation of at least a position or rotation with respect to the second image performed by a user so that a point cloud drawn in the second image matches a shape of an object drawn in the first image (Zhang, see pars. [0036] Referring to FIG. 1B, an example scene 110 shown in two images 112 and 114 is shown from two different perspectives whether captured from a single camera that is moved from image to image or captured from multiple cameras each providing one of the images 112 or 114. As shown, each image 112 and 114 has its own overlap region 118 and 116 respectively. By one form, the overlap region is cuboid, and more particularly cubic where the edge length is the same in all three directions (width, height, and length of the overlap region). It will be appreciated, however, that the methods herein could be applied to overlap regions with other shapes than cuboid or cubic. Also, and as described below, the methods herein modify the position and edge length of the overlap regions. Herein the position refers to the center point of the overlap region and the edge length is the edge forming a corner of the overlap region where two planes of the overlap region meet. It will be understood, however, that other alternatives could be used such as a corner or other desired location in or on the overlap region to be used as the position discussed herein, and the edge length (also referred to herein as edge size) may be referred to as edge dimension as well, but could be replaced with any overlap dimension that conveniently and accurately indicates a size of the overlap region for the purposes explained herein.
[0037] As mentioned, the point cloud registration may be achieved by a principal axes alignment within or on the overlap region. Such principal axes alignment is represented by a principal axes pair having two detected principal axes 120 and 122 that are detected by using the points in the point clouds. Herein, the principal axes refers to a Cartesian coordinate system with three axes (x, y, z), and the principal axes may be referred to herein as a principal coordinate system or principal axes system. Thus, the term “principal axes pair” or “candidate principal axes pair” discussed below refers to two matched principal axes each with three axes (x, y, z) and where the x-axis of one principal axes in the pair is matched to the x-axis of the other principal axes of the pair, and so on for each of the three axes. The principal axes 120 and 122 here are shown aligned to each other as well as respectively aligned to the overlap regions 116 and 118. The details of such alignment is provided below.).
Motivation
Regarding claim 3. The information processing device according to claim 2, wherein the adjustment operation includes an adjustment operation for a plurality of places included in the second image (Zhang, see at least par. [0046] A best principal axes pair of the set of candidate principal axes pairs then may be iteratively matched to individual overlap region samples, and this represents the posterior probability mentioned above. Determining the best match may comprise differencing an individual overlap region sample varied from the first or second overlap region, and differenced from the other of the initial first or second overlap region maintained with an initial center point position but where an edge length is varied to match the edge length of the overlap region sample being compared to. Before the differencing is performed, the overlap region sample may be adjusted to align with the principal axes of one of the candidate principal axes pair, and the other initial overlap region (with its center point fixed and edge length equal to the edge length of the overlap region sample) may be adjusted to align with the other principal axes of the one candidate principal axes pair. By one form, the differencing comprises determining a hamming difference between binary histograms point distributions within the adjusted overlap region sample and the initial overlap region. The candidate principal axes pair causing the smallest hamming distance associated with an overlap region sample, among all of the candidate principal axes pairs, may be considered a match to the overlap region sample.).
Claims 5-6, 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1), as applied claim 4 above and further in view of CRONVALL et al. (US 20250292596 A1)
Regarding claim 5. Minear in view of Chen discloses the information processing device according to claim 4, but Minear in view of Chen does not discloses wherein the point cloud data is a set of key frames including a group of feature point data extracted from a captured image, the geographic coordinate information, and virtual space position information. However,
CRONVALL discloses:
wherein the point cloud data is a set of key frames including a group of feature point data extracted from a captured image, the geographic coordinate information, and virtual space position information (CRONVALL, see pars. [0069-0073).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the point cloud data is a set of key frames including a group of feature point data extracted from a captured image, the geographic coordinate information, and virtual space position information, as provided by CRONVALL. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to provid[ing] an efficient multi-stage (e.g., two-stage) multi-view object detection, such as three-dimensional (3D) object detection. In one or more examples, the systems and techniques extract high resolution patches (e.g., regions) from input images based on 3D detections, and use these patches at full (e.g., high) resolution for object detection. (CRONVALL, see par. [0033]).
Regarding claim 6. Minear in view of Chen and further in view of CRONVALL discloses the information processing device according to claim 5, and Minear in view of Chen and further in view of CRONVALL further discloses wherein the control unit generates the second image from the set of key frames (CRONVALL, see at least par. [0071] In some cases, the 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from certain input images from the image sensor 202 (and/or other camera) to the SLAM map. For example, 6DoF SLAM can use feature point associations from an input image to determine the pose (position and orientation) of the image sensor 202 and/or system 200 for the input image. 6DoF mapping can also be performed to update the SLAM map. In some cases, the SLAM map maintained using the 6DoF SLAM can contain 3D feature points triangulated from two or more images. For example, key frames can be selected from input images or a video stream to represent an observed scene. For every key frame, a respective 6DoF camera pose associated with the image can be determined. The pose of the image sensor 202 and/or the system 200 can be determined by projecting features from the 3D SLAM map into an image or video frame and updating the camera pose from verified 2D-3D correspondences.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the control unit generates the second image from the set of key frames, as provided by CRONVALL. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to provid[ing] an efficient multi-stage (e.g., two-stage) multi-view object detection, such as three-dimensional (3D) object detection. In one or more examples, the systems and techniques extract high resolution patches (e.g., regions) from input images based on 3D detections, and use these patches at full (e.g., high) resolution for object detection. (CRONVALL, see par. [0033]).
Regarding claim 9. Minear in view of Chen and further in view of CRONVALL discloses the information processing device according to claim 5, and Minear in view of Chen and further in view of CRONVALL further discloses wherein the control unit updates the geographic coordinate information included in each of the key frames on a basis of a result of the matching operation (CRONVALL, see pars. [0071-0072]).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the control unit updates the geographic coordinate information included in each of the key frames on a basis of a result of the matching operation, as provided by CRONVALL. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to provid[ing] an efficient multi-stage (e.g., two-stage) multi-view object detection, such as three-dimensional (3D) object detection. In one or more examples, the systems and techniques extract high resolution patches (e.g., regions) from input images based on 3D detections, and use these patches at full (e.g., high) resolution for object detection. (CRONVALL, see par. [0033]).
Regarding claim 18. Minear in view of Chen discloses the information processing device according to claim 1, but Minear in view of Chen does not discloses wherein as the geographic coordinate information of the point cloud data, geographic coordinate information of an imaging point of a captured image from which each piece of feature point data is extracted is used as an initial state. However,
CRONVALL discloses:
wherein as the geographic coordinate information of the point cloud data, geographic coordinate information of an imaging point of a captured image from which each piece of feature point data is extracted is used as an initial state (CRONVALL, see at least par. [0072] In one illustrative example, the compute components 210 can extract feature points from certain input images (e.g., every input image, a subset of the input images, etc.) or from each key frame. A feature point (also referred to as a registration point) as used herein is a distinctive or identifiable part of an image, such as a part of a hand, an edge of a table, among others. Features extracted from a captured image can represent distinct feature points along three-dimensional space (e.g., coordinates on X, Y, and Z-axes), and every feature point can have an associated feature location. [0073] In some cases, sensor data, such as images captured by the image capture and processing system 100 of FIG. 1, point clouds captured by LIDAR/RADAR sensors, etc., may be processed by neural networks and/or machine learning (ML) systems.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein as the geographic coordinate information of the point cloud data, geographic coordinate information of an imaging point of a captured image from which each piece of feature point data is extracted is used as an initial state, as provided by CRONVALL. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to provid[ing] an efficient multi-stage (e.g., two-stage) multi-view object detection, such as three-dimensional (3D) object detection. In one or more examples, the systems and techniques extract high resolution patches (e.g., regions) from input images based on 3D detections, and use these patches at full (e.g., high) resolution for object detection. (CRONVALL, see par. [0033]).
Claims 7 is rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1), further in view of CRONVALL et al. (US 20250292596 A1), as applied claim 6 above, and further in view of ZHAO et al. (US 20210056751 A1).
Regarding claim 7. Minear in view of Chen and further in view of CRONVALL discloses the information processing device according to claim 6, but Minear in view of Chen and further in view of CRONVALL does not disclose wherein the control unit generates the second image by converting a position of each feature point included in the group of the feature point data of each of the key frames from screen coordinates of each captured image into world coordinates and drawing a point cloud on a horizontal plane. However,
ZHAO discloses:
wherein the control unit generates the second image by converting a position of each feature point included in the group of the feature point data of each of the key frames from screen coordinates of each captured image into world coordinates and drawing a point cloud on a horizontal plane (Zhao, see at least par. [0020] and [0048],, the method for converting local coordinates of the 3D model of a single room into global coordinates includes: enabling the photo capture unit to move a predetermined distance, and obtaining, by the capture position acquisition unit, coordinates of two endpoints of the predetermined distance, where a ratio of the difference between the coordinates of the two endpoints to the predetermined distance is the scale of the local coordinates to the global coordinates; or estimating, by using one or more feature points identified by the capture position acquisition unit, a ratio of the height of the plane of the floor or ceiling of the space to the actual height of the photo capture unit, to obtain the scale of the local coordinates to the global coordinates. identifying one or more image areas of at least one of a floor, a ceiling, and a wall in the image based on a deep learning method; and (S32) dividing the identified image area(s) into blocks based on an image processing technology, where each block is approximately considered as one plane, image blocks of the floor and the ceiling are located on a horizontal plane, and an image block of the wall is located on a vertical plane; and generating the 3D model by solving an equation for each plane, where for two planes that intersect in the image, an intersecting line of the two planes is used as a constraint, so that an error between a calculated intersecting line and an actually observed intersecting line is minimized.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the control unit generates the second image by converting a position of each feature point included in the group of the feature point data of each of the key frames from screen coordinates of each captured image into world coordinates and drawing a point cloud on a horizontal plane, as provided by ZHAO. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to support multiple photo capture devices, and automatically assemble 3D models of various photo capture points based on an relative position of each photo capture point and capture direction information of a camera lens that are obtained during photo capture, to generate an overall 3D model. In the present disclosure, a 2D floorplan can also be generated. (ZHAO, see par. [0013]).
Claims 8 is rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1), further in view of CRONVALL et al. (US 20250292596 A1), as applied claim 5 above, and further in view Speasl et al. (US 20050129385 A1).
Regarding claim 8. Minear in view of Chen and further in view of CRONVALL discloses the information processing device according to claim 5, but Minear in view of Chen and further in view of CRONVALL does not disclose wherein the control unit calculates latitude and longitude of an origin of the second image, and superimposes and displays the second image such that the calculated latitude and longitude match latitude and longitude of the first image. However,
Speasl discloses:
wherein the control unit calculates latitude and longitude of an origin of the second image, and superimposes and displays the second image such that the calculated latitude and longitude match latitude and longitude of the first image (Speasl, see at least par. [0059], Following each calculation, latitude and longitude data points are plotted on top of the user selected map database using a video overlay feature incorporated into the Display Controller 6. The precise location of the plot is determined by comparing the latitude and longitudinal range of the map database and calculating Cartesian-type (X-Y) coordinates for the display 5. The plot can be refreshed each time a new geolocation calculation is completed resulting in a different display coordinate set.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the control unit calculates latitude and longitude of an origin of the second image, and superimposes and displays the second image such that the calculated latitude and longitude match latitude and longitude of the first image, as provided by Speasl. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to facilitate a successful outcome. Collaboration can be accomplished by simply handing the device to the meeting participants for subsequent dialogue with their colleagues or for copying onto their PC, docking stations, or other display devices, for further study and commentary (Speasl, see par. [0010]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1), further in view of CRONVALL et al. (US 20250292596 A1), further in view Speasl et al. (US 20050129385 A1), as applied claim 8 above, and further in view of Abhiram et al. (US 20190287297 A1).
Regarding claim 10. Minear in view of Chen, further in view of CRONVALL and further in view of Speasl discloses the information processing device according to claim 8, but Minear in view of Chen, further in view of CRONVALL and further in view of Speasl does not disclose wherein the control unit calculates and updates latitude and longitude of each of the key frames corresponding to an arbitrary position of the second image after adjustment on a basis of a shift amount of a position or a rotation of the second image by the matching operation. However,
Abhiram discloses:
wherein the control unit calculates and updates latitude and longitude of each of the key frames corresponding to an arbitrary position of the second image after adjustment on a basis of a shift amount of a position or a rotation of the second image by the matching operation (Abhiram, see FIG. 2 and pars. [0018] An internal depth map representation may be an array of distances or map coordinates corresponding to each pixel in a scene frame or with respect to any other reference coordinate system (e.g., a three-dimensional point cloud representation). The convolver processor 206 may be used to estimate regional optical flow as a basis for ego-motion tracking (also known as visual odometry). The optical flow between the current scene frame 204 and a matching key frame 212 is the shifting of corresponding pixels between the key frame and the current scene frame. As part of the optical flow determination, convolver processor 206 may be used to map each input scene frame 204 to a lower-dimensional feature space, in which distance between feature vectors represents visual dissimilarity. The computed feature vectors are used for key frame (KF) matching and loop closure. Computing optical flow enables tracking and identifying moving objects in the environment. Optical flow representation 210 may be, for example, an array of vectors corresponding to the change in position for each pixel in a scene frame. Simultaneous localization and mapping (SLAM) algorithms are applied to a depth map representation 208, an optical flow representation 210, and one or more key frames 212 to estimate the transformation with six degrees of freedom (position (x, y, z), and orientation (roll, pitch, yaw)) between the key frame 212 and the current scene frame 204. Each camera 100 of system 200 provides the depth, position, and orientation estimates for each scene frame via a system interface 214 to other components of a distributed system, e.g., for combining information from multiple cameras 100 into an aggregated three-dimensional environment model 216, the model including objects associated with identity and movement metadata..).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the control unit calculates and updates latitude and longitude of each of the key frames corresponding to an arbitrary position of the second image after adjustment on a basis of a shift amount of a position or a rotation of the second image by the matching operation, as provided by Abhiram. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to provide technical solutions by using the optimized convolver processor, these determinations are fast and updates can be provided in real-time (Abhiram, see par. [0019]).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1), further in view of CRONVALL et al. (US 20250292596 A1), as applied claim 5 above, and further in view of Newell et al. (US 20230314171 A1).
Regarding claim 11. Minear in view of Chen, further in view of CRONVALL discloses the information processing device according to claim 5, but Minear in view of Chen, further in view of CRONVALL does not discloses wherein the control unit calculates and updates latitude and longitude of each of the key frames on a basis of a result of the matching operation performed on a plurality of places of the second image. However,
Newell discloses:
wherein the control unit calculates and updates latitude and longitude of each of the key frames on a basis of a result of the matching operation performed on a plurality of places of the second image (Newell, see pars. [0029] Images may be judged to be similar based upon many different metrics, for example similarity by color, texture, or other recognizable content such as faces. This concept can be extended to portions of images or Regions of Interest (ROI). The query can be either a whole image or a portion (ROI) of the image. The images retrieved can be matched either as whole images, or each image can be searched for a corresponding region similar to the query. In the context of the current invention, CBIR may be used to automatically select assets that are similar to some other automatically selected asset. Scene classifiers identify or classify a scene into one or more scene types (e.g., beach, indoor, etc.) or one or more activities (e.g., running, etc.). Example scene classifiers are listed at 404. A face detector 405 is used to find as many faces as possible in image collections. Face recognition 406 is the identification or classification of a face to an example of a person or a label associated with a person based on facial features. Face clustering uses data generated from detection and feature extraction algorithms to group faces that appear to be similar. Location-based data 407 can include cell tower locations, GPS coordinates, and network routers. All image and video media have an associated location that may or may not be metadata archived with the image or video file. These are typically stored with the image as metadata by the recording device, which captures an image or sound. Location-based data can be very powerful when used in concert with other attributes for media clustering. Item 408 exemplifies identification or classification of a detected event into a semantic category such as birthday, wedding, etc. Media in an event are associated with the same setting or activity per a unit of time and preferably related to the subjective intent of the user or group of users. Media in a sub-event have similar content within an event. Within each event, media can also be clustered into separate groups of relevant content called sub-events. Media in an event are associated with same setting or activity, while media in a sub-event have similar content within an event. An image value index, or "IVI", 409 is defined as the degree of importance (significance or usefulness/utility) that an individual user might associate with a particular asset. IVI algorithm development could utilize image features such as: sharpness and quality, camera-related metadata (exposure, time, date), image understanding (skin or face detection and size of skin/face area), and behavioral measures (viewing time, magnification, editing, printing, sharing). Video key frame extraction 410 is the process of extracting key-frames and/or salient shot, scene, or event, and the associated audio to provide a summary or highlight of a video sequence. ).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the control unit calculates and updates latitude and longitude of each of the key frames on a basis of a result of the matching operation performed on a plurality of places of the second image, as provided by OHASHI. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to create[ion] of various video-style special effects including fades and dissolves, image-collaging, backgrounds and borders, and colorization makes the presentation much more interesting to the viewer and can greatly enhance the emotional content of the images being presented. The proliferation in the home of new television-based viewing platforms able to accommodate multimedia, including DVD and Video CD players, also increases the demand for this type of presentation. (Nevell, see par. [0002]).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Minear et al. (US 20100207936 A1) in view of Chen et al. (US 20160187486 A1), further in view of CRONVALL et al. (US 20250292596 A1), and further in view of Newell et al. (US 20230314171 A1), as applied claim 11 above, and further in view Kobayashi et al.
(US 20180310066 A1).
Regarding claim 12. Minear in view of Chen, further in view of CRONVALL discloses the information processing device according to claim 11, but Minear in view of Chen, further in view of CRONVALL does not disclose wherein the control unit calculates and updates, using linear interpolation, the latitude and longitude of one or more key frames located between the key frames corresponding to the plurality of places where the matching operation has been performed. However,
Kobayashi discloses:
wherein the control unit calculates and updates, using linear interpolation, the latitude and longitude of one or more key frames located between the key frames corresponding to the plurality of places where the matching operation has been performed (Kobayashi, see par. [0110] The method of setting item areas associated with frames other than the key frames through interpolation may be a method in which each coordinate is calculated through linear interpolation of coordinates of two points (the so-called linear interpolation), and other than this method, may be a method that employs image analysis. For example, in the key frame A, the metadata creation terminal 7 extracts an image of a rectangular area (equivalent to an item area) that is defined by the coordinates (XA1, YA1) and (XA2, YA2) input by the operator, and extracts features amounts (for example, color information, or the like) of the image in the rectangular area. Similarly, in the key frame B, the metadata creation terminal 7 extracts an image of a rectangular area that is defined by the coordinates (XB1, YB1) and (XB2, YB2) input by the operator, and extracts feature amounts of the image in the rectangular area. Then, the metadata creation terminal 7 may extract, from each frame between the key frames A and B, an image within a range from two feature amounts extracted from the key frame A and from the feature amounts extracted from the key frame B (e.g., an image within a predetermined color range), and on the basis of the rectangular area surrounding the extracted image, the metadata creation terminal 7 may interpolate item area setting data associated with that frame.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method and apparatus of Minear, to have wherein the control unit calculates and updates, using linear interpolation, the latitude and longitude of one or more key frames located between the key frames corresponding to the plurality of places where the matching operation has been performed, as provided by Kobayashi. The modification provides an improved system and method for superimposing and displaying a virtual object on a real space in a state where a user can visually recognize the real space in front of the user, thereby to achieving the aforementioned object is to cause a moving image reproduction device including a display unit and an input unit, to display a moving image in the display unit by displaying frames included in moving image data in the display unit while changing the frames from one to the next sequentially; set item areas while changing from one to the next as the frames are changed from one to the next sequentially, the item areas being set in advance to the frames; and when one of the item areas that is set in a corresponding one of the frames that is being displayed in the display unit is selected using the input unit, display item information associated with the item area in the display unit.. (Kobayashi, see par. [0010]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KIM THANH THI TRAN whose telephone number is (571)270-1408. The examiner can normally be reached Monday-Friday 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ALICIA HARRINGTON can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KIM THANH T TRAN/Examiner, Art Unit 2615
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
Prosecution Insights