DETAILED ACTION
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wong et al. (Publication: US 2021/0004933 A1) in view of Barna et al. (Publication: US 2017/0223808 A1) and Hagiwara et al. (Publication: US 2015/0193416 A1).
Regarding claim 1, Wong discloses an electronic apparatus comprising: a camera; a memory; and a processor configured to ([0182], Fig. 2 - The system and method of the preferred embodiment and variations thereof can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions are preferably executed by computer-executable components preferably integrated with the system. The computer-readable medium can be stored on any suitable computer-readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component is preferably a general or application specific processor, but any suitable dedicated hardware or hardware/firmware combination device can additionally or alternatively execute the instructions.):
obtain an image, captured through the camera, corresponding to a specific region and at least one device ([0058] Each set of images is preferably of a predetermined quality (e.g. measured by image characteristics, level of accuracy, etc.). Predetermined quality can relate to the level of accuracy in which the visual sensors of the system capture, process, store, compress, transmit, and display signals that form an image but can additionally or alternatively relate to any other suitable elements that can function to process images. Image quality is preferably maintained by taking multiple images of the same region of a scene, using automatic features of a visual sensor to measure and adjust characteristics of an image (e.g., white balance, exposure, noise, focus, etc.), but additionally or alternatively include using manual feature of a visual sensor to measure and adjust characteristics of an image, or by any other suitable for method for ensuring sufficient quality.);
obtain information corresponding to the specific region and identification information and arrangement information for the at least one device included in the image based on the image (Figure 7 – shape and wall is recognized via camera as wall in Fig. 7 .
[0137] - coarsely aligning images is achieved by the initial computation of 2D correspondences (e.g., by matching features, floor, at S211) between multiple images for use in warp computation. semantic segmentation can be used to find correct feature correspondences as shown in FIG. 7, where matching features must also share containing or surrounding classes. semantic segmentation can be used to find correct feature correspondences as shown in FIG. 7, where matching features must also share containing or surrounding classes.
[0176] - the blending can additionally or alternatively include image normalization. Cropping can include making the final panorama rectangular for the desired horizontal and vertical field of view (e.g., according to a predetermined size, shape, etc.), dimension of a region, height and width must be know in order to crop to the desired area.
[0064] – camera took area of interest such as floor. [0096] - semantic segmentation performed on an image of a room can segment the image into a region assigned a “door” semantic label and a region assigned a “table” semantic label. Each line feature of the “door” region can be identified and labeled (e.g., “top edge of door”, “bottom edge of door”, “left edge of door, “right edge of door”, etc.). Each line feature of the “table” region can be identified and labeled in a similar manner. In this manner, line features in the two images match if their semantic segmentation contexts also match. For example, lines that each have a “left edge of first door” label match, whereas a line having a “left edge of door” label and a line having a “right edge of door” label do not match.);); and provide a map corresponding to the arrangement information for the at least one device in the specific region based on the information corresponding to the specific region, the identification information and the arrangement information for the at least one device ([0010] FIG. 3 is a preferred embodiment of S100 and S200, where the set of images in S100 contains five images, the set of images are aligned to the center image's plane, camera pose estimates and three-dimensional and two-dimensional correspondences are found and used for coarse alignment in S200.
[0096] - determining a match between the first and second features includes accessing a semantic segmentation context for each feature and comparing the semantic segmentation context of the features. Semantic segmentation context can be generated by performing semantic segmentation on the first and second images. Performing semantic segmentation for an image can include segmenting the image into multiple regions, and assigning a semantic label to each region, and optionally a semantic label to one or more features of the region. For example, semantic segmentation performed on an image of a room can segment the image into a region assigned a “door” semantic label and a region assigned a “table” semantic label. Each line feature of the “door” region can be identified and labeled (e.g., “top edge of door”, “bottom edge of door”, “left edge of door, “right edge of door”, etc.). Each line feature of the “table” region can be identified and labeled in a similar manner. In this manner, line features in the two images match if their semantic segmentation contexts also match. For example, lines that each have a “left edge of first door” label match, whereas a line having a “left edge of door” label and a line having a “right edge of door” label do not match. ).
Wong does not Barna discloses
a user interface ([0104] – user changes the virtual mapping with the user interface elements 726 and 728 to the right by touching and dragging a particular scene to reposition it in the list);
based on a user request being input through the user interface, obtain an image ([0104] – user changes the virtual mapping with the user interface elements 726 and 728 to the right by touching and dragging a particular scene to reposition it in the list).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wong with a user interface; based on a user request being input through the user interface, obtain an image as taught by Barna . The motivation for doing is to provide convenience as taught by Barna.
Wong in view of Barna do not however Hagiwara discloses
the map is provided to be displayed as geometric representations without the
image captured through the camera ([0065] When the user requests to edit the floor map (step S801), the user is presented with a user interface screen displaying the floor map (step S802) as illustrated in FIG. 9B. The floor map displays a typical office space which contains a group of cubicles on the top left of the user interface screen, a group of small offices on the top right of the user interface screen, a large office on the bottom right of the user interface screen and a large empty room on the bottom left of the user interface screen. In addition, the application 101a further presents to the user options for placing icons and comments on the floor map (step S803). Next, the terminal receives a request from the user to place an icon onto the floor map (step S804) and displays a user interface screen to the user as shown in FIG. 9C.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wong in view of Barna with the map is provided to be displayed as geometric representations without the image captured through the camera as taught by Hagiwara. The motivation for doing is to optimize use of digital maps as taught by Hagiwara.
Regarding claim 2, Wong in view of Barna, Hagiwara disclose all the limitation of claim 1.
Wong discloses obtain a panorama image of the specific region and the at least one device by panorama scan through the camera (
[0023] - FIG. 12A is a variant of S100 where on-device AR tracking capabilities are used to assign AR camera poses to each image for a panorama.
[0052] - first set of images could capture to a cooking space (e.g., kitchen, commercial kitchen, kitchenette, cookhouse, galley, etc.) and a second set of images could capture a communal space (e.g., living area, work area, dining area, lounge, reception area, etc.). ).
Regarding claim 3, Wong in view of Barna, Hagiwara disclose all the limitation of claim 2.
Won discloses obtain curve information of the panorama image ([0042] – images are captures include edges.);
and obtain information associated with a shape of the specific region by recognizing at least one of a corner of the specific region, a height of the specific region, and a width of the specific region based on the obtained curve information of the panorama image (Figure 7 – shape and wall is recognized via camera as wall in Fig. 7 .
[0137] - coarsely aligning images is achieved by the initial computation of 2D correspondences (e.g., by matching features, floor, at S211) between multiple images for use in warp computation. semantic segmentation can be used to find correct feature correspondences as shown in FIG. 7, where matching features must also share containing or surrounding classes. semantic segmentation can be used to find correct feature correspondences as shown in FIG. 7, where matching features must also share containing or surrounding classes.
[0176] - the blending can additionally or alternatively include image normalization. Cropping can include making the final panorama rectangular for the desired horizontal and vertical field of view (e.g., according to a predetermined size, shape, etc.), dimension of a region, height and width must be know in order to crop to the desired area.).
Regarding claim 4, Wong in view of Barna, Hagiwara disclose all the limitation of claim 3.
Wong discloses based on the curve information of the panorama image, as a region showing a first curvature in the specific region, obtain an actual width of the region to be smaller than a width of the region which is included in the panorama image ([0176] S400 preferably processes the locally-aligned images from S300, but can additionally or alternatively process any other suitable data. S400 can include cropping, or otherwise modifying the images. Cropping can include making the final panorama rectangular for the desired horizontal and vertical field of view.); and
based on the curve information of the panorama image, as a region showing a second curvature in the specific region, obtain the actual width of the region to be greater than the width of the region which is included in the panorama image (
[0042] – images are captures include edges.
[0176] S400 preferably processes the locally-aligned images from S300, but can additionally or alternatively process any other suitable data. S400 can include blending. Blending can include removing any visible edges when compositing the seam-carved images and/or blend pixels from overlapping images “greater”. Blending can be done in the image domain, the gradient domain, the frequency domain, or other formulations. The blending can additionally or alternatively include image normalization.
[0144] Aligning images finely S300 can include finely aligning images in the set of coarsely aligned images to generate a set of finely aligned images. Aligning images can include finely aligning pairs of images in the set of coarsely aligned images (e.g., pairs of adjacent images, etc.) (e.g., generated at S200). S300 can include for at least one image in the set of coarsely aligned images, accessing constraints S310, dividing the image into a uniform grid mesh S320, and adjusting grid mesh vertices to maximize a combination of constraint values S330. In variants, the image is modified using the adjusted grid mesh vertices (S340). Constraint values can include at least a feature constraint value and a global shape anchor constraint value. In some variations, adjusting vertices includes generating one or more sets of adjusted vertices.).
Regarding claim 5, Wong in view of Barna, Hagiwara disclose all the limitation of claim 1.
Wong discloses obtain an augmented reality image of the specific region and the at least one device by augmented reality scan through the camera ([0052] - first set of images could capture to a cooking space (e.g., kitchen, commercial kitchen, kitchenette, cookhouse, galley, etc.) and a second set of images could capture a communal space (e.g., living area, work area, dining area, lounge, reception area, etc.).
[0023] - FIG. 12A is a variant of S100 where on-device AR tracking capabilities are used to assign AR camera poses to each image for a panorama.).
Regarding claim 6, Wong in view of Barna, Hagiwara disclose all the limitation of claim 5.
Wong discloses obtain information associated with a corner of the specific region by performing the augmented reality scan of the corner of the specific region through the camera ([0064] – camera took area of interest such as floor. [0096] - semantic segmentation performed on an image of a room can segment the image into a region assigned a “door” semantic label and a region assigned a “table” semantic label. Each line feature of the “door” region can be identified and labeled (e.g., “top edge of door”, “bottom edge of door”, “left edge of door, “right edge of door”, etc.). Each line feature of the “table” region can be identified and labeled in a similar manner. In this manner, line features in the two images match if their semantic segmentation contexts also match. For example, lines that each have a “left edge of first door” label match, whereas a line having a “left edge of door” label and a line having a “right edge of door” label do not match.);
based on information of a floor of the specific region being scanned through the camera, obtain information associated with an area of the floor and a size and a shape of the specific region based on a pattern of the floor (
[0064] - guided capture can include instructing the user to capture meaningful features, such as the floor and/or dominant wall and/or ceiling, and/or wall-floor seam, within the image frame. In an eleventh example, guided capture can include instructing the user via auditory instructions.
[0164] - a 3D location of at least one point feature detected in the image whose vertices are being adjusted is identified (e.g., by using photogrammetry based on the set of captured images and the estimated camera pose data). Each point feature is projected onto an image plane of the adjacent image and the pano image plane by using at least the associated 3D location. For each point feature detected in the image whose vertices are being adjusted, to make the warped feature point position close to its projection in the pano image plane, a score is calculated that identifies whether the point feature in the image whose vertices are being adjusted is aligned with the corresponding projection in the pano image plane after the vertices have been adjusted. Such 3D location constraint is mainly applied on the non-overlapping regions when warping a new image into the pano image. Warping “size and a shape”.
); and
obtain the information associated with the size and the shape of the specific region based on a route of the camera while the image corresponding to the specific region is captured and a shape of a wall recognized through the camera (
Figure 7 – shape and wall is recognized via camera as wall in Fig. 7 .
[0137] - coarsely aligning images is achieved by the initial computation of 2D correspondences (e.g., by matching features, floor, at S211) between multiple images for use in warp computation. semantic segmentation can be used to find correct feature correspondences as shown in FIG. 7, where matching features must also share containing or surrounding classes. semantic segmentation can be used to find correct feature correspondences as shown in FIG. 7, where matching features must also share containing or surrounding classes.).
Regarding claim 7, Wong in view of Barna, Hagiwara disclose all the limitation of claim 6.
Wong discloses to generate a map at a visual angle of a user by a corner rotation method for setting a random point at a center of the specific region obtained as the augmented reality image and rotating the specific region or the at least one device in the obtained image around a virtual axis that vertically passes through the floor of the specific region obtained as the augmented reality image while passing through the random point (See Fig. 11 .
[0142] - aligning images globally is achieved using a coarse warp for coarse alignment can be calculated from device rotations, “generate a map at a visual angle”
[0056] Each set of images is preferably oriented about an axis of rotation for ease of user capture. The axis of rotation is preferably the vertical or horizontal axis through the camera lens, the vertical or horizontal axis through the capture device body, or the vector representing gravity. However, the images can be additionally or alternatively oriented in any other suitable rotation, “random point”. Axis is the center.
[0067] - using gyro rotations to augment the last seen AR pose estimate.).
Regarding claim 8, Wong in view of Barna, Hagiwara disclose all the limitation of claim 7.
Wong discloses wherein the user request is a first user request and the map is a first map and the processor is configured to: based on the at least one device being scanned through the camera, perform control to output information associated with the at least one device by identifying the at least one device based on pre-trained data ([0010] FIG. 3 is a preferred embodiment of S100 and S200, where the set of images in S100 contains five images, the set of images are aligned to the center image's plane, camera pose estimates and three-dimensional and two-dimensional correspondences are found and used for coarse alignment in S200.
[0096] - determining a match between the first and second features includes accessing a semantic segmentation context for each feature and comparing the semantic segmentation context of the features. Semantic segmentation context can be generated by performing semantic segmentation on the first and second images. Performing semantic segmentation for an image can include segmenting the image into multiple regions, and assigning a semantic label to each region, and optionally a semantic label to one or more features of the region. For example, semantic segmentation performed on an image of a room can segment the image into a region assigned a “door” semantic label and a region assigned a “table” semantic label. Each line feature of the “door” region can be identified and labeled (e.g., “top edge of door”, “bottom edge of door”, “left edge of door, “right edge of door”, etc.). Each line feature of the “table” region can be identified and labeled in a similar manner. In this manner, line features in the two images match if their semantic segmentation contexts also match. For example, lines that each have a “left edge of first door” label match, whereas a line having a “left edge of door” label and a line having a “right edge of door” label do not match.
[0137] - joint line detection and matching can be achieved using deep learning. sparse correspondences can be densified (filling holes) using optimization, filtering, machine learning, or ensemble techniques, “pre-trained” based on panorama.);
and corresponding to the output information being input, provide a second map corresponding to the specific region based on identification information and arrangement information for the at least one device (
[0013] FIG. 5B is an example of coarse warping from S200 and fine warping from S300.
[0143] - S300 functions to attempt to locally correct remaining image misalignments after coarse alignment is complete (e.g. locally moving, floating, or stretching local areas of the image to better align with other images), as shown in FIG. 5a and FIG. 5b (e.g., using constrained local warping, content-preserving warps, global shape anchors, etc.). S300 is preferably performed after S200, but can additionally or alternatively be performed.
S300 is based on S200 that includes the area of the identifiable and alignment information .
[0149] - S300 can be hierarchical (e.g. more detail in some areas, less in areas of uniform appearance, etc.). In a preferred example, the vertices of the mesh cells are mapped to the final panorama coordinates.).
Barna discloses based on a second user request corresponding to the output information perform an operation ([0104] - change the virtual mapping with the user interface elements 726 and 728 to the right by touching and dragging a particular scene to reposition it in the list. Several users operated on the interface as shown in Fig. 1.) .
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wong in view of Barna, Hagiwara with based on a second user request corresponding to the output information perform an operation as taught by Barna . The motivation for doing is to provide convenience as taught by Barna.
Regarding claim 9, Wong in view of Barna, Hagiwara disclose all the limitation of claim 1.
Wong discloses to output a map, and wherein the output map comprises at least one of information associated with a temperature of the specific region, a quality of air in the specific region, and a lighting of the specific region ([0005] - A mobile or other user computer device is connected to the wide area network and has a graphical user interface enabling a user to access the server control software to control and configure the lighting fixtures associated with wireless devices at the site according to the user's granted permissions, including virtually mapping user interface elements of user control devices to specific lighting effects. By using a user computer device to configure user control devices, configuration can advantageously be done without physical access to either the user control device or to the lighting controllers, lighting fixtures, or other devices at the installation site to be associated with the user control device.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wong in view of Barna, Hagiwara with to output a map, and wherein the output map comprises at least one of information associated with a temperature of the specific region, a quality of air in the specific region, and a lighting of the specific region as taught by Barna . The motivation for doing is to provide convenience as taught by Barna.
Regarding claim 10, Wong in view of Barna, Hagiwara disclose all the limitation of claim 1.
Wong discloses wherein the map comprises an outline showing information associated with a size and a shape of the specific region and an icon showing arrangement information of the at least one device ([0167] Global shape constraints are forms of geometry preservation constraints that can include ensuring that the locally refined warped image maintain a similar shape as the initial global warping as shown in FIG. 5c (e.g., from S200), and can include penalizing the border mesh vertices from drifting away from initial locations. The initial locations can be the vertices' locations post-coarse adjustment (e.g., post-S200), be the vertices' location pre-coarse adjustment, be a target vertex location, or be any other suitable location. The initial locations can be relative to the image being warped, the reference image, a 3D coordinate system, a 2D coordinate system, and/or any other suitable reference frame. For example, enforcing global shape constraints can include enforcing the locally refined warp to maintain a similar shape as the initial global warping output by S200, but can additionally or alternatively include any other suitable constraints. Warp changes the size and shape.), and
wherein the processor is configured to correct the information associated with the size and the shape of the specific region based on [[user's dragging of the outline]] of the specific region, and correct the arrangement information of the at least one device based on [[user's dragging]] of the icon (
[0148] - S300 warping, shape changes, based on S200.
[0147] - The data received by S300 to be processed is preferably the set of images, any 2D & 3D feature correspondences, and computed coarse warps (e.g. set of coarsely aligned images obtained at S200)
).
Barna discloses correction is based on user's dragging of the outline of the specific region (0104] - shown in FIG. 17, the list of scenes to the left can be reordered to change the virtual mapping with the user interface elements 726 and 728 to the right by touching and dragging a particular scene to reposition it in the list, thereby changing the virtual mapping to what is then displayed.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Wong in view of Barna, Hagiwara with correction is based on user's dragging of the outline of the specific region as taught by Barna . The motivation for doing is to provide convenience as taught by Barna.
Regarding claim 11, see rejection on claim 1.
Regarding claim 12, see rejection on claim 2.
Regarding claim 13, see rejection on claim 3.
Regarding claim 14, see rejection on claim 4.
Regarding claim 15, see rejection on claim 5.
Response to Arguments
Examiner suggests to amend a specific element in the claim that when reading a claim in light of the invention, it directs to a unique technology.
Claim Rejection Under 35 U.S.C. 103
Applicant asserts “Applicant respectfully notes that claims 1 and 11 were amended, in the Amendment submitted on August 14, 2025, to recite the map of those claims including an icon corresponding to the identification information and the arrangement information for the at least one device. Applicant further respectfully maintains that the cited prior art neither disclose nor suggests such a feature.”
Examiner disagrees.
Wong discloses provide a map corresponding to the arrangement information for the at least one device in the specific region based on the information corresponding to the specific region, the identification information and the arrangement information for the at least one device. Please see the rejection above for detail.
Applicant asserts “Applicant is of the understanding, per the Interview conducted on December 18, 2025, that amended claim 1 patentably distinguishes over the currently cited references. Applicant once again expresses appreciation to the Examiner for the valuable communication during the Interview.”
Examiner agrees. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Hagiwara reference.
Regarding dependent claims 2 – 10 and 12 - 15, the Applicant asserts that they are not obvious over based on their dependency from independent claim 1 and 11 respectively. The examiner cannot concur with the Applicant respectfully from same reason noted in the examiner’s response to argument asserted from claim 1 and 11 respectively.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Wu whose telephone number is (571) 270-0724. The examiner can normally be reached on Monday-Thursday and alternate Fridays (9:30am - 6:00pm) EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ming Wu/
Primary Examiner, Art Unit 2618