DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 32- 62 are presented for examination based on the amendment filed on 06/18/2026.
Claim 32 is objected, for minor issue.
Claims 32- 62 are rejected under 35 U.S.C. 112(b).
Claims 32- 62 are rejected under 35 U.S.C. 112(a).
Claims 32-62 are not found eligible under 35 USC 101.
Claims 32-41, 43-45, 52-55 and 58 -62 are rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), further in the view of Bichu; Tanmay Nitin (US 20210082086 A1).
Claims 42 and 50 rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1) further in the view of CubiCasa FAQ & Manual, retrieved on March 26, 2019, from hllps://www.cubi.casa/faq/, 5 pages.
Claim 46 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Adsumilli; Balineedu Chowdary (US 20190387167 A1).
Claim 47 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Fathi; Habib (US 11106911 B1).
Claim 48 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Williams, Don, and Peter D. Burns. "Image Stitching: Exploring Practices, Software, and Performance." Archiving Conference. Vol. 10. Society of Imaging Science and Technology, 2013.
Claim 49 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Williams, Don, and Peter D. Burns. "Image Stitching: Exploring Practices, Software, and Performance." Archiving Conference. Vol. 10. Society of Imaging Science and Technology, 2013 further in the view of Adsumilli; Balineedu Chowdary (US 20190387167 A1)
Claim 51 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view Bhogal; Nikhil (US 20120293607 A1)
Claim 56 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view Norio Yamamoto (JPH11213143A).
Claim 57 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view GANI HAR (US 20210232719 A1).
This action is final rejection.
Response to Arguments
The objection to the drawing is withdrawn, since applicant specifies the predicted amount of time for producing the mapping information and the amount of time to complete building floor plan are the same as shown on fig 2F-2K.
The 35 U.S.C 101 rejection is not withdrawn, since the claim limitation recites a mental process without any additional element which is significantly more, See 35 U.S.C 101 rejection below.
The 112 rejection is withdrawn based on the amended claim, but a new 112 rejection is recited on the claim, see U.S.C 112 rejection below.
Applicants Argument: Applicant’s arguments directed the 103 rejections are based on newly amended subject matter.
Examiner’s Response: All arguments are addressed in the 103 rejections of the claims below.
Applicant’s argument: Applicant respectfully submits that none of the cited references, alone or in combination, teaches or suggests "outputting, to an image acquisition device, feedback that includes the predicted amount of time and at least one recommendation for reducing the predicted amount of time," as recited in claim 32.
Examinations response: The examiner disagrees since the modified model of Li- Pham_ Wang - Bichu teaches all the limitations of claim 32, as it is cited below in U.S.C 103 rejection, Pham teaches displaying of the amount of time it take to stitch image as a graph and it also recommends using of high overlap image to reduce the amount of time (Section Ill, Additionally, it is recommended that aerial images be captured with at least 80%-85% overlap to obtain the best output quality for forest or densely vegetated areas), while visual attributes are thought by Li on [0010] “borders between adjacent walls; borders between walls and a floor; borders between walls and a ceiling;” , while Wang teach estimating the amount of time it takes to stitch two images with resolution (which is a visual attribute, according specification [0015]), so by combining those, it would be obvious for a person of ordinary skill in the art to output the predicted amount of time and feedback to image acquisition device. See U.S.C 103, rejection below for more.
Claim Objections
Claim 32 is objected to because of the following informalities:
“One or mor of the rooms”, should be one or more rooms in claim 32.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 32- 62 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
As of claim 32, it recites “at least one recommendation for reducing the
predicted amount of time” and “modifying the predicted amount of time for generating the mapping information based on visual attributes of the at least one additional image”, this recites one additional image’s visual attribute is used and the time is modified (reduced, since it gives feedback including a recommended to reduce the amount of time), How does adding additional images visual attribute reduce the amount of time? while on claim 47, it time is reduced by replacing on of the images, since the specification does not appear to disclose, How adding additional images visual attribute reduce the amount of time?, it makes the claim unclean.
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 32-62 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention, as it is outlined above, in 112(b), claim 32 recites reducing of the amount of time by adding additional images visual attribute and the specification does not disclose How adding additional images visual attribute reduce the amount of time?, and as it is shown on Fig 2F- 2I, the amount of time increase as the quality increase. (MPEP 2161.01 states: For instance, generic claim language in the original disclosure does not satisfy the written description requirement if it fails to support the scope of the genus claimed. Ariad, 598 F.3d at 1349-50, 94 USPQ2d at 1171 ("[A]n adequate written description of a claimed genus requires more than a generic statement of an invention’s boundaries.") (citing Eli Lilly, 119 F.3d at 1568, 43 USPQ2d at 1405-06); Enzo Biochem, Inc. v. Gen-Probe, Inc., 323 F.3d 956, 968, 63 USPQ2d 1609, 1616 (Fed. Cir. 2002) (holding that generic claim language appearing in ipsis verbis in the original specification did not satisfy the written description requirement because it failed to support the scope of the genus claimed); Fiers v. Revel, 984 F.2d 1164, 1170, 25 USPQ2d 1601, 1606 (Fed. Cir. 1993) (rejecting the argument that "only similar language in the specification or original claims is necessary to satisfy the written description requirement").
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 32 -62 are rejected under 35 U.S.C. 101 because the claim invention
recites a judicial exception, which is directed to judicial exception of an abstract idea, as
it has not been integrated into practical application, and the claim further does not recite
significantly more than the judicial exception.
Step 1: claims 32- 60 are directed to a system, which is a machine, so it is under a statutory category of invention. Claim 61 is directed to a method, which is a process, so it is under a statutory category of invention. Claim 61 is directed to a non-transitory computer readable medium, which is a manufacture, so it is a statutory category of the invention.
Step 2A: Prong 1: Yes, the claims recite abstract idea. Claims 32- 62 recites abstract
ideas, which falls under a mathematical concept/ mental process since a human mind can observe image to assess visual attribute and based on the visual attribute a human mind can also make judgment on how long it may take for generating mapping information of the building. Abstract ideas are bolded as shown below.
Claims 32, and 61-62:
assessing one or more visual attributes of the images that affect an automated generation process for generating mapping information for the
building: under its broadest reasonable interpretation, this claim limitation recites a mental process. In this limitation there is no specific way of assessing a visual attribute is recited so, a human mind can assess a visual attribute of images by performing observation/judgment on the obtained image. According to [0015] quality of image can be a visual attribute, and a human mind can observe the image and make judgment about its quality. Therefore, this claim recites abstract idea. See MPEP 2106.04(a)(2)(iii).
generating, based on the one or more visual attributes, a predicted amount of time for generating the mapping information; under its broadest reasonable interpretation this limitation recites a mental process. This limitation does not recite any specific way of predicting amount of time so a human mind can quantitatively judge how long a mapping generation might take based on the visual attribute. Therefore, this claim recites abstract idea. See MPEP 2106.04(a)(2)(iii), and if prediction of time is done mathematically it will be considered as a mathematical concept. See MPEP 2106.04(a)(2)(1)(c).
modifying the predicted amount of time for generating the mapping information based on visual attributes of the at least one additional image; under its broadest reasonable interpretation this limitation recites a mental process. A human mind can perform a quantitative judgment on the amount of time based on the additional image visual attribute. A human mind can make observations on the additional image visual attribute and by using the previous amount of time as a base, a human mind can make a modified quantitative judgment. Therefore, this claim recites abstract idea. See MPEP 2106.04(a)(2)(iii).
Step 2A Prong 2: No
The above judicial exceptions do not recite additional elements that integrate
the exceptions into a practical application of the exception because the claims do not
have additional elements of a combination of additional elements that apply, rely on or
use the judicial exception in a manner that imposes a meaningful limit on the judicial
exception.
Claims recite gathering data and outputting information which is insignificant extra solution activity. Adding insignificant extra-solution activity to the judicial exception, e.g., mere data gathering in conjunction with a law of nature or abstract idea such as a step of obtaining information about credit card transactions so that the information can be analyzed by an abstract mental process, as discussed in CyberSource V. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011) (see MPEP § 2106.05(g), and claims also recites data manipulation by “displaying” outputs - Selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display, Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354-55, 119 USPQ2d 1739, 1742 (Fed. Cir. 2016); MPEP 2106.05(g).
The claim limitations which recite data gatherings, outputting and manipulation are listed below.
Claims 32, and 61-62:
obtaining images of rooms of a building, each of the images depicting
at least one wall of one or mor of the rooms and having been captured at one
of a plurality of acquisition locations in the building- (insignificant extra- solution activity – data gathering such as 'obtaining information'. See MPEP 2106.05(g).).
outputting, to an image acquisition device, feedback that includes the
predicted amount of time and - (insignificant extra-solution activity – data gathering,
such as 'outputting data'. See MPEP 2106.05(g).) at least one recommendation for reducing the predicted amount of time (under MPEP 2106.05(f) this could be an idea of a solution or outcome i.e. the claim fails to recite details on how a solution to a problem is accomplished. The recitation of the claim limitation attempts to cover a solution (recommendation to reduce the predicted amount of time) to a problem without restriction on how the result is accomplished and no description of the mechanism for accomplishing the result, so it does not integrate into a practical application or provide significantly more because this type of recitation is equivalent to the word “apply it").
responsive to outputting the feedback, obtaining at least one additional image from the image acquisition device, the at least one additional image depicting the one or more of the rooms and being captured at an acquisition location other than the plurality of acquisition locations; - (insignificant extra- solution activity – data gathering such as 'obtaining information'. See MPEP 2106.05(g).).
outputting, to the image acquisition device, revised feedback that
includes the modified predicted amount of time. - (insignificant extra-solution activity – data gathering, such as 'outputting data'. See MPEP 2106.05(g).)
Step 2B: No:
The claims do not cite additional elements which are significantly more than the
abstract idea. As outlined above the claims merely use a computer as a tool to gather
and output information for the abstract idea. As claim 32, and 61-62, they recite a computer component and a non-transitory computer readable medium, which are used as a tool to execute the abstract idea of assessing visual attribute, predicting amount of time and inputting/outputting data. Using a computer as a tool is not significantly more since there is no improvement to the computer other than using computer components to store, process and display analyzed information.
Generally, the independent claims recite abstract ideas based on the
above analysis and let’s see if there are any significant more claim limitations exist for
the dependent claims, since the dependent claims inherit abstract ideas from independent claim.
Claim 33:
wherein the image acquisition device includes one or more cameras used to capture the images and the at least one additional image – additional element of a camera is recited but is used as a tool to gather information (capture image), so the additional element is not significantly more
Claim 34:
wherein the images comprise panorama images having 360 degrees of horizontal visual coverage around a vertical axis- it further defines the obtained data, so there is no additional element which is significantly more.
Claim 35:
wherein the panorama images are in a spherical format having an equirectangular projection n- it further defines the obtained data, so there is no additional element which is significantly more.
Claim 36:
wherein the mapping information comprises a floor plan for the building - it further narrows the abstract idea, so there is no additional element which is significantly more
Claims 37:
wherein the feedback and the revised feedback are output in a graphical user interface displayed via the image acquisition device- - (insignificant extra-solution activity – data gathering, such as 'outputting data'. See MPEP 2106.05(g).)
Claim 38:
wherein the one or more visual attributes include at least one of lighting, motion blur, or presence of distracting elements in the images- further defines abstract idea, so there is no additional element which is significantly more.
Claim 39:
wherein the one or more visual attributes include at least one of an amount of inter-image overlap between the images or uncertainty in an estimated room shape - further defines abstract idea, so there is no additional element which is significantly more.
Claim 40:
wherein the feedback comprises a quality classification assessment for each of the images – further defines abstract idea since a human mind can make judgment of the image quality by making observation, so there is no additional element which is significantly more.
Claim 41:
the operations further comprising determining, for one or more of the rooms, an estimated room shape of the room based on the images, wherein the feedback further includes the estimated room shape – it recites a mental process of estimating room shape, since a human mind can make observation and can tell the room shape based on the image, so there is no additional element which is significantly more.
Claim 42:
wherein the predicted amount of time is generated based at least in part on room shape uncertainties associated with the estimated room shape- it further defines abstract idea, so there is no additional element which is significantly more.
Claim 43:
wherein the feedback comprises a partial version of the mapping information generated from the images – further defines output data, so there is no additional element which is significantly more.
Claim 44:
wherein assessing the one or more visual attributes includes identifying one or more problems to correct, and wherein the at least one recommendation for reducing the predicted amount of time includes information to correct the one or more problems - it further defines abstract idea of assessing visual attribute, and a mental process of recommendation, so there is no additional element which is significantly more.
Claim 45:
wherein the one or more problems include at least one of a lack of overlap between visual data of at least one of the images and at least one other image, a lack of coverage of the rooms, or uncertainty in determination of a room shape of at least one of the rooms – further defines the abstract idea, so there is no additional element which is significantly more.
Claim 46:
wherein the at least one recommendation for reducing the predicted amount of time includes automated instructions to adjust one or more settings of the image acquisition device – it further defines the mental process of recommendation, and a human mind can write instruction to make the device automatic setting change, so there is no additional element which is significantly more.
Claim 47:
wherein the at least one additional image replaces one of the images, insignificant extra-solution activity – data gathering such as 'obtaining information'. See MPEP 2106.05(g).
and wherein the modified predicted amount of time is less than the predicted amount of time – further define the abstract idea, so there is no additional element which is significantly more.
Claim 48:
the operations further comprising, before obtaining the images, generating an initial prediction of the amount of time based at least in part on computing resources available for the automated generation process and on publicly available information about the building – it is a mental process of observing the computing device and check the available amount of computational resource and using its predict the amount of time, so there is additional element of a computing device but it is also used as a tool, so there is no additional element which is significantly more.
Claim 49:
wherein the initial prediction is further based at least in part on information describing at least one of a historical performance of a user associated with the image acquisition device or a priority level associated with the user- it further defines the abstract idea of predicting time based on available resource, so there is no additional element which is significantly more.
Claim 50:
wherein the predicted amount of time is further generated based on one or more of a size of the building, a number of the rooms of the building, or a style of the building – it further defines the abstract idea of predicting time based on any of the above inputs, so there is no additional element which is significantly more.
Claim 51:
the operations further comprising: determining a current status of acquiring images for the automated generation process, the current status including at least one of an amount completed or an amount of the images remaining to be acquired; and – this recites a mental process which can be performed through observation/ judgment, so there is no additional element which is significantly more.
outputting the current status together with the feedback - insignificant extra-solution activity – data gathering, such as 'outputting data'. See MPEP 2106.05(g).
Claim 52:
transmitting the images and the at least one additional image to one or more
server computing devices; and - insignificant extra solution activity- WURC, MPEP 2106.05(d)(II) by at least "I. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321
causing the one or more server computing devices to generate a floor plan for the building using the images and the at least one additional image by determining final room shapes for the rooms and combining the final room shapes – this claim recites an additional element of a computing device, but it is used as a tool to generate a floor plan, and a floor plan can be made by a human mind using a pen and paper, so there is no additional element which is significantly more.
Claim 53:
wherein generating the floor plan comprises obtaining user input related to at least one of the final room shapes or combining the final room shapes - insignificant extra-solution activity – data gathering such as 'obtaining information'. See MPEP 2106.05(g).
Claim 54:
wherein the images are captured: without using depth information from depth-sensing sensors describing distances to surrounding surfaces; and – it further specifies a criterion to gather information, so there is no additional element which is significantly more.
using one or more inertial measurement unit sensors of the image acquisition
device to acquire motion data – it recites additional element of a sensor, but it is used as a tool to gather information, so there is no additional element which is significantly more.
Claim 55:
determining, for each of the images, acquisition pose information based at least in part on the motion data – this claim limitation recites a mental process since a human can determine the pose information using a physical aid and by performing observation/ judgment, so there is no additional element which is significantly more.
Claim 56:
wherein the predicted amount of time is generated using a regression model- it further defines the abstract idea of predicting amount of time using a mathematical concept, so there is no additional element which is significantly more.
Claim 57:
wherein the mapping information includes at least one of a three-dimensional model of an interior of the building or a virtual tour involving user-selectable inter connections between at least some of the images: - this claim limitation further specifies the type of information the mapping information includes, so there is no additional element which is significantly more.
Claim 58:
wherein the mapping information is useable to control navigation of one or more mobile devices – this claim limitation further defines the ideal use, so there is no additional element which is significantly more.
Claim 59:
wherein assessing the one or more visual attributes includes identifying structural features of the rooms including at least one of borders between adjacent walls, doorways, windows, or non-doorway wall openings – it further defines the abstract idea of assessing visual attribute, so there is no additional element which is significantly more.
Claim 60:
wherein the at least one recommendation for reducing the predicted amount of time includes at least one of a recommendation to recapture one of the images or a recommendation to capture an image at a different acquisition location- it recites a mental process of a user recommendation, so there is no additional element which is significantly more.
Generally, based on the above claim by claim analysis and claims as a whole, it
does not recite any additional element which is significantly more than the claimed invention. The claim is merely a mental process of predicting amount of time using based on visual attribute and outputting feedback including recommendation to reduce amount of time using a computer and merely using a computer is not significantly more than abstract idea since there is no improvement in the computer recited.
Therefore claims 32-62 are not found eligible under 35 USC 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 32-41, 43-45, 50, 52-55 and 58 -62 are rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG(CA 3090629 A 1 ), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu(US 20160198088 A1), further in the view of Bichu; Tanmay Nitin (US 20210082086 A1).
As of claim 32, Li teaches one or more processors; and a computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
([0058], Accordingly, embodiments of the present disclosure may be practiced with other computer system configurations.
obtaining images of rooms of a building, each of the images depicting at least one wall of one or mor of the rooms and having been captured at one of a plurality of acquisition locations in the building;( [0009], In at least some embodiments, the defined area includes an interior of a multi-room building (e.g., a house, office, etc.), and the generated information includes a floor map of the
building, such as a 2D (two- dimensional) overhead view (e.g., an orthographic top
view) of a schematic floor map that is generated from an analysis of multiple 360 degree
spherical panorama images acquired at various viewing locations within the building
(e.g., using an image acquisition device with a spherical camera having one or more
fisheye lenses to capture a panorama image that extends 360 degrees horizontally
around a vertical axis).
assessing one or more visual attributes of the images that affect an automated generation process for generating mapping information for the building; ([0010], As one non-exclusive example, one or more MIGM system operator users may, in at least some embodiments, manipulate information overlaid on a displayed acquired image of a room and/or a generated shape of the room (e.g., a 2D overhead floor map) in order to identify structural and other visual features of the room in which the image was acquired, such as to identify one or more of the following: borders between adjacent walls; borders between walls and a floor; borders between walls and a ceiling;windows and/or sky-lights; passages into and/or out of the room, such as doors and other openings in walls, stairs, etc.; other structures (e.g., represented as cuboid shapes), such as countertops, bath tubs, sinks, fireplaces, and furniture; etc).
the at least one additional image depicting the one or more of the rooms and being captured at an acquisition location other than the plurality of acquisition locations; ([0012], such additional information may further include, for example, exterior dimensions and/or shape of the building, additional images and/or annotation information acquired corresponding to particular locations within the building (optionally
for locations different from viewing locations of the acquired panorama or other images) …[0012], As one example, one or more types of additional information about a building may be received and associated with the floor map (e.g., with particular locations in the floor map), such as additional images, textual and/or audio annotations or other descriptions of particular rooms or other locations, other audio information, such as recordings of ambient noise; overall dimension information, etc.).
While Li does not explicitly teach generating, based on the one or more visual attributes, a predicted amount of time for generating the mapping information; outputting, to an image acquisition device, feedback that includes the predicted amount of time and at least one recommendation for reducing the predicted amount of time; responsive to outputting the feedback, obtaining at least one additional image from the image acquisition device, modifying the predicted amount of time for generating the mapping information based on visual attributes of the at least one additional image; and outputting, to the image acquisition device, revised feedback that includes the modified predicted amount of time.
While Pham teaches outputting, to an image acquisition device, feedback that includes the predicted amount of time and at least one recommendation for reducing the predicted amount of time; (Abstract The proposed method improves the visual quality of the stitched image, by decreasing the estimated reprojection error and the number of observed visual distortions. In addition, the proposed method can substantially reduce the processing time compared with conventional stitching methods ... Section iv, c, The simulations were performed on a Windows 10 computer equipped with an Intel Core i7-10700F CPU, 32GB RAM, Section Ill, Additionally, it is recommended that aerial images be captured with at least 80%-85% overlap to obtain the best output quality for forest or densely vegetated areas [35], [36]. If the input images are too dense or several images share the same overlapping region, the dense correspondence may lead to misregistration [8]. Consequently, the stitching performance of densely overlapped input images becomes inadequate). The output is shown as a graph on figure 12 as it is shown below. As it is shown on figure 12 the graph shown a predicted amount of time to stitch different number of images.
outputting, to the image acquisition device, revised feedback that includes the modified predicted amount of time (
PNG
media_image1.png
677
798
media_image1.png
Greyscale
As shown above on the graph , for different image number it have updated amount of time it takes for stitching using different model and the proposed model use image attributes like number of images and resolution. (Section 4, The overall processing time of the proposed method for stitching 100, 150, and 200 images of resolution 5472 _ 3648 is 67.1, 106.5, and 135.7 (seconds), respectively), as shown in the graph the second number of images(20) is considered as the second images with attribute if the number of images.
Li and Pham are considered to be analogous to the claimed invention since they focus on automatic operation of involved image analysis and combining images to create a complete scene. Therefore it would be obvious to try for a person of ordinary skill in the art before the effective filing date to combine Pham teaching of estimating amount of time based on different attributes( while predicting the amount of time based on visual attribute is thought below by “Wang”), in to Li 's model in order to further provide benefits in allowing improved automated navigation of a building by mobile devices (e.g., semi-autonomous or fully autonomous vehicles), including to significantly reduce their computing power used and time used to attempt to otherwise learn a building's layout (Li, para 13) and improves the visual quality of the stitched image, by decreasing the estimated reprojection error and the number of observed visual distortions. In addition, the proposed method can substantially reduce the processing time compared with conventional stitching methods(Pham, abstract).
The modified model does not explicitly teach generating, based on one or more visual attributes, a predicted amount of time for generating the mapping information; responsive to outputting the feedback, obtaining at least one additional image from the image acquisition device, modifying the predicted amount of time for generating the mapping information based on visual attributes of the at least one additional image.
While Wang teaches generating, based on the one or more visual attributes, a predicted amount of time for generating the mapping information; ([0315], The time required may depend on the resolution of the images, the processing power of the processors, and other factors. For example, in an embodiment, the time required to stitch two images each with a resolution of 1920*1080 is at most 0.2 second where the stitching is implemented by a 600 MHz ARM Cortex A9 processor).
responsive to outputting the feedback, obtaining at least one additional image from the image acquisition device, ([0285], Subsequently, at or around time t2, the image capturing device captures a second image 804. The rotation may stop briefly (e.g., for 0.1 second, 0.2 second, 0.5 second, or 1 second) when an image is captured so as to improve the quality of the image. Alternatively, the rotation may be continuous and the images are taken while the image capturing device is rotating. At the time the second image 804 is captured, the image capturing device has rotated β1 degrees since the previous image, where in can be any positive number).
Wang is considered to be analogous to the claimed invention since, it teaches estimating amount of time it takes to stitch images based on visual attributes. Therefore it would be obvious for a person of ordinary skill in the art , before the effective filling date to integrate Wang’s teaching of estimating time based on visual attribute and obtain a second image, after it gets feedback so that the second image is improved in quality, into the modified model to output the estimated time as a graph as the modified model teaches.
The motivation would have been the motion damping applied by the damping elements may be used to stabilize the load, thereby improving the quality of images captured by the load (e.g., image capturing device), as well as reducing the computational complexity of image stitching steps required to generate a panoramic image based on the captured images (Wang, [0255]).
The modified model of Li- Pham-Wang does not explicitly teach modifying the predicted amount of time for generating the mapping information based on visual attributes of the at least one additional image.
While Bichu teaches modifying the predicted amount of time for generating the mapping information based on visual attributes of the at least one additional image; and ([359], In some embodiments as described above, to increase computational speed and decrease the amount of computation required, the processes of rectification, disparity map generation, disparity map filtering, alignment including disparity correction, stitching …In one example, application of this approach reduces the total computation time for image or video stitching by more than a factor of two (e.g., from 55.67 seconds to 25 seconds)…[0363], the stitched result in the quarter resolution case appears sufficient depending on the application. With the reduction in number of pixels ( 1/16th times the number of pixels) at a quarter resolution, the total time required for image stitching was brought down from 25 s to 16.3 s).
Bichu is considered to be analogous to the claimed invention since image stitching and factors that affect the amount of time to stitch. Therefore it would be obvious for a person of ordinary skill in the art before the effective filing date to integrate Bichu teaching of modifying the predicted amount of time to stich images into the modified model to modify the amount of time for mapping based on recommendation like having more overlap, reducing the number of pixels to reduce the amount of time for generating mapping information.
The motivation would have been an adjustment process is performed after the filtering process to adjust the filtered disparity maps before the alignment process in order to improve the stitched image of the complete scene to form a single 360-degree view (Bichu, [0259]).
Claims 61 and 62 are in the same scope as claim 32, so claim 61 and 62 are rejected under the same rational as claim 32.
As of claim 33, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the image acquisition device includes one or more cameras used to capture the images and the at least one additional image ([0020] While such a mobile image acquisition device may include various hardware components, such as a camera, one or more sensors (e.g., a gyroscope, an accelerometer, a compass, etc., such as part of one or more IMUs, or inertial measurement units, of the mobile device; an altimeter; light detector; etc.)…[0026], Figure 2B continues the example of Figure 2A, and illustrates an additional perspective image 250b taken in a northwesterly direction from viewing location 210B in the living room of house 198 of Figure 1 B).
As of claim 34, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the images comprise panorama images having 360 degrees of horizontal visual coverage around a vertical axis ([0020], 20, Figure 1 B depicts a block diagram of an exemplary building interior environment in which 360° spherical panorama images are generated, for use by the MIGM system to generate and provide a corresponding building floor map, as discussed in greater detail with respect to Figures 2A-2W, as well as for use in presenting the panorama images to users. In particular, Figure 1 B illustrates one story of a multi-story building 198 with an interior that was captured at least in part via multiple panorama images, such as by a mobile image acquisition device 185 with image acquisition capabilities as it is moved through the building interior to a sequence of multiple viewing locations 210 (e.g., starting at viewing location 21 0A, moving to viewing location 21 OB along travel path 115,etc.).
As of claim 35, the modified model teaches all the limitations of claim 34, and Li also teaches wherein the panorama images are in a spherical format having an equirectangular projection ( [0028], Figure 20 continues the examples of Figures 2A-2C, and illustrates a 360° spherical panorama image 250d (e.g., taken from viewing location 210B), which displays the entire living room in an equirectangular format).
As of claim 36, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the mapping information comprises a floor plan for the building ([0065], In the example of Figures 5A-5C, the generated mapping information includes a floor map of a building (e.g., a house), but in other embodiments, other types of mapping information may be generated for other types of buildings and used in other manners, as discussed elsewhere herein).
As of claim 37, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the feedback and the revised feedback are output in a graphical user interface displayed via the image acquisition device ([0072], The user also moves the corresponding wall visual indicators in the seventh GUI portion to show those walls on which the corresponding visual horizontal indicators in the sixth GUI portion indicate horizontal spaces. As the user manipulates the visual horizontal indicators in the sixth GUI portion and/or the wall visual indicators in the room shape in the seventh GUI portion, the information in both GUI portions is updated to reflect the user manipulations (e.g., simultaneously with the user manipulations) and show the user changes that are made. The image and the specified horizontal visual indicators are then further analyzed to determine the height of the camera within the room when the image was taken, such as based on the angle between horizontal visual indicators on two adjacent walls.
As of claim 38, the modified model teaches all the limitations of claim 32, and Bichu teaches wherein the one or more visual attributes include at least one of lighting, motion blur, or presence of distracting elements in the images (Bichu, Abstract, the disclosed image stitching technique includes depth or disparity estimation, alignment, and blending processes configured to be computationally efficient and produce quality results by limiting the presence of noticeable seams and artifacts in the final stitched image… [0006] But as is often the case in actual real-world situations, the captured scene includes rapid movement or dynamic motion resulting in artifacts due in part to time differences between image segments or changes in intensity across the captured scene). , Examiner Note: As it is cited above artifacts are considered as distracting elements
As of claim 39, the modified model teaches all the limitations of claim 32, and Wang teaches wherein the one or more visual attributes include at least one of an amount of inter-image overlap between the images or uncertainty in an estimated room shape (Wang [0033]- [0036], In some embodiments, the image capturing device is configured to capture the plurality of images such that there is at least a 50% overlap between adjacent images of the plurality of images).).
As of claim 40, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the feedback comprises a quality classification assessment for each of the images ([0061] In addition, the routine may further optionally monitor the motion of the mobile device in some embodiments during movement to the next viewing location, and provide one or more guidance cues to the user regarding the motion of the mobile device, quality of the sensor data and/or video information being captured, associated lighting/environmental conditions, advisability of capturing a next viewing location, and any other suitable aspects of capturing the linking information). Examiner note: as it is cited above Li teaches providing one or more guidance (feedback) to the user about quality of the video information being captured, so since the quality of the video information is known, its quality is a quality classification assessment.
As of claim 41, the modified model teaches all the limitations of claim 32, and Li also teaches the operations further comprising determining, for one or more of the rooms, an estimated room shape of the room based on the images ([0029] –[0030], The second GU I pane 255e displays an initial room shape 260e corresponding to the visual border and corner GU I controls 280 and 285 in the first GUI pane, such that changes to the room shape in the second GUI pane or to the visual border and corner GUI controls 280 and 285 in the first GUI pane cause corresponding changes in the other pane ... After the corner and border visual representations are displayed in the GUI, the user is able to manipulate the visual representations to match corresponding features of the room that are visible in the underlying panorama image, as illustrated in further detail with respect to Figure 2F (feedback)… [46] As in Figures 2E-2H, a second GUI pane 255s is shown with an initial room shape 260s corresponding to the visual border and corner GUI controls 280, 282 and 285 in the first GUI pane, Label 581 Receive selection of room with defined room shape and image from within room, optionally automatically determine likely image location in room from image analysis and/or metadata, and display GUI with image in sixth GUI portion and room shape in seventh GUI portion) wherein the feedback further includes the estimated room shape ([0068], Once the user is done, the final room shape in the second GUI portion provides a user-defined estimate of the room shape).
As of claim 43, the modified model teaches all the limitations of claim 41, and Li also teaches wherein the feedback comprises a partial version of the mapping information generated from the images ([0012] a more detailed floor plan of the building may further be generated, such as with sufficient detail to allow blueprints or other architectural plans to be generated. In addition, if estimated size information includes height information from floors to ceilings, a 30 (three-dimensional) model (e.g., with full height information represented) and/or 2.50 (two-and-a-half dimensional) model (e.g., with partial representations of height shown) of some or all of the 20 (two-dimensional) floor map may be created (optionally with information from in room images projected on the walls of the models),
As of claim 44, the modified model teaches all the limitations of claim 32, and Li also teaches wherein assessing the one or more visual attributes includes identifying one or more problems to correct, ([0062], For example, the ICA system may provide one or more notifications to the user regarding the information acquired during capture of the multiple viewing locations and optionally corresponding linking information, such as if it determines that one or more segments of the recorded information are of insufficient or undesirable quality, or do not appear to provide complete coverage of the building) and wherein the at least one recommendation for reducing the predicted amount of time includes information to correct the one or more problems ([0040], As illustrated in the example of the seventh GUI pane of Figure 20, however, the initial relative layout of the room shapes may not be precisely aligned at first, such as with room shape 239 being at a slight angle relative to that of room 238 (e.g., due to imperfections in the locations and/or connections of the inter-room openings). While such layout problems may be immediately adjusted by the user and/or by the MIGM system in an automated manner (e.g., by applying layout constraints and/or optimizations, such as that room shapes connect at parallel walls) in some embodiments and situations...,[0062] After block 425, the routine continues to block 435 to optionally preprocess the acquired 360° spherical panorama images before their subsequent use for generating related mapping information, such as to perform an equirectangular projection for each such image, such that straight vertical data (e.g., the sides of a typical rectangular door frame, a typical border between 2 adjacent walls).
As of claim 45, the modified model teaches all the limitations of claim 44, and Pham also teaches wherein the one or more problems include at least one of a lack of overlap between visual data of at least one of the images (section B. “hybrid method for multiple aerial image stitching”, In this subsection, we present the proposed hybrid method for multiple aerial image stitching. First, we describe a fast feature detection and matching method using the determined image overlapping regions. Second, we briefly review projective warp for non-overlapping regions) and Li also teaches at least one other image, a lack of coverage of the rooms, or uncertainty in determination of a room shape of at least one of the rooms ([0062], For example, the ICA system may provide one or more notifications to the user regarding the information acquired during capture of the multiple viewing locations and optionally corresponding linking information, such as if it determines that one or more segments of the recorded information are of insufficient or undesirable quality, or do not appear to provide complete coverage of the building).
As of claim 52, the modified model teaches all the limitations of claim 32, and Li also teaches the operations further comprising: transmitting the images and the at least one additional image to one or more server computing devices; and ([0053], The server computing system(s) 300 and executing ICA system 340, and server computing system(s) 380 and executing MIGM system 389, may communicate with each other and with other computing systems and devices in this illustrated embodiment via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), such as to interact with user client computing devices 390 (e.g., used to view floor maps, and optionally associated images and/or other related information) and on fig 3 from label 300 and 360 to 390 by using a network 399) causing the one or more server computing devices to generate a floor plan for the building using the images and the at least one additional image by determining final room shapes for the rooms and (fig 5A label 530 optionally apply automated optimization(s) to room shape based on geometrical constraints and/or other information, optionally determine room dimensions from one or more known or estimated lengths, and store final room shape along with other determined/received information, [0045], After all of the room shape layout information has been specified and any such wall width information has been determined, the final results may be used to generate a floor map of the house… [0068] Once the user is done, the final room shape in the second GUI portion provides a user-defined estimate of the room shape…[0079] a floor plan may be generated from the floor map that includes dimension information for the rooms and the overall building, [claim 14] determining, by the one or more computing devices and for each of the one or more rooms, a final user-defined room shape for the room by combining information received from the user about borders of the room from each of the multiple images taken within the room) combining the final room shapes ([0057], the routine continues to block 545 to determine if the instructions received in block 505 (or subsequently) indicate to combine room shapes of multiple indicated rooms in a building to determine their relative layout).
As of claim 53, the modified model teaches all the limitations of claim 52, and Li also teaches wherein generating the floor plan comprises obtaining user input related to at least one of the final room shapes or combining the final room shapes ([0045], After all of the room shape layout information has been specified and any such wall width information has been determined, the final results may be used to generate a 20 floor map of the house ...[0068] Once the user is done, the final room shape in the second GUI portion provides a user-defined estimate of the room shape. As noted elsewhere herein, in some embodiments the routine may further perform one or more types of automated modifications to the user defined room shape estimate to determine the final defined room shape, while in other embodiments the user-defined room shape estimate may be used as the final defined room shape, Figure 5A label 535 ... [79] a floor plan may be generated from the floor map that includes dimension information for the rooms and the overall building).
As of claim 54, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the images are captured: without using depth information from depth-sensing sensors describing distances to surrounding surfaces; and ([0009], analysis of multiple 360° spherical panorama images acquired at various viewing locations within the building .... in at least some such embodiments, the generating of the mapping information is further performed without having or using information acquired from depth-sensing equipment about distances from the images' viewing locations to walls or other objects in the surrounding building) using one or more inertial measurement unit sensors of the image acquisition device to acquire motion data ([0020], While such a mobile image acquisition device may include various hardware components, such as a camera, one or more sensors (e.g., a gyroscope, an accelerometer, a compass, etc., such as part of one or more IMUs, or inertial measurement units, of the mobile device; an altimeter; light detector; etc.), a GPS receiver, one or more hardware processors, memory, a display, a microphone, etc., label 422 Optionally obtain acceleration data and/or other linking information generated by image acquisition device as it moves to next viewing location in sequence).
As of claim 55, the modified model teaches all the limitations of claim 54, and Li also teaches determining, for each of the images, acquisition pose information based at least in part on the motion data ([0072] The image and the specified horizontal visual indicators are then further analyzed to determine the height of the camera within the room when the image was taken, such as based on the angle between horizontal visual indicators on two adjacent walls, [73] ... The image and the specified vertical visual indicators are then further analyzed to triangulate the location of the camera within the room when the image was taken, such as based on the distance in the image between vertical visual indicators relative to their locations on the walls of the room shape).
As of claim 58, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the mapping information is useable to control navigation of one or more mobile devices ([0009], The generated floor map and/or other generated mapping-related information may be further used in one or more manners in various embodiments, such as for controlling navigation of mobile devices (e.g., autonomous vehicles), for display on one or more client devices in corresponding GUls (graphical user interfaces), etc.).
As of claim 59, the modified model teaches all the limitations of claim 32, and Li also teaches wherein assessing the one or more visual attributes includes identifying structural features of the rooms including at least one of borders between adjacent walls, doorways, windows, or non-doorway wall openings ([0010], in order to identify structural and other visual features of the room in which the image was acquired, such as to identify one or more of the following: borders between adjacent walls; borders between walls and a floor; borders between walls and a ceiling; windows and/or sky-lights; passages into and/or out of the room, such as doors and other openings in walls, stairs, etc.).
As of claim 60, the modified model teaches all the limitations of claim 32, and Li also teaches wherein the at least one recommendation for reducing the predicted amount of time includes at least one of a recommendation to recapture one of the images or a recommendation to capture an image at a different acquisition location ([ 0079] Other information about the building may also be retrieved from, for example, one or more external sources (e.g., online databases, 'crowd-sourced' information provided by one or more end users, etc.), and associated with and linked to the floor map and/or particular locations within the floor map - such additional information may further include, for example, exterior dimensions and/or shape of the building, additional images and/or annotation information acquired corresponding to particular locations within the building (optionally for locations different from viewing locations of the acquired panorama or other images), etc.).
Claims 42 and 50 are rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1) further in the view of CubiCasa FAQ & Manual, retrieved on March 26, 2019, from hllps://www.cubi.casa/faq/, 5 pages.
As of claim 42, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 41, but it does not explicitly teach the limitations of claim 42.
While CubiCasa teaches wherein the predicted amount of time is generated based at least in part on room shape uncertainties associated with the estimated room shape (FAQs, 2o How long it takes to take a video from a 2000 sq ft home which 1 s in 2 floors? Capturing process takes from 4-6, minutes in 2000sq ft space. capturing time depends on the area and complexity of the indoor space. 41. How long does H: take to build a floorplan of a 4-bed house? The scanning speed can be fairly fast, so 1000 sqft should not take more than 5 minutes.) as listed above room shape uncertainty is mapped to the area and complexity of the indoor space and size).
Cubicasa is considered to be analogous to the claimed invention since it focuses on floor plan generation. Therefore it would be obvious to a person of ordinary skill in the art before the effective filling date to combine Cubicasa teaching of providing updated amount of time it takes based on space and complicity for one room and 4 room, to the modified model in order to increase the accuracy of generation process with the error margin of around 5%(Cubicasa, FAQ).
As of claim 50, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 32, but it does not explicitly teach the limitation of claim 50.
While CubiCasa teaches wherein the predicted amount of time is further generated based on one or more of a size of the building, a number of the rooms of the building, or a style of the building (FAQ, 41. How long does it take to build a floorplan of a 4-bed house? The scanning speed can be fairly fast, so 1000sqft should not take more then 5 minutes).
Cubicasa is considered to be analogous to the claimed invention since it focuses on floor plan generation. Therefore it would be obvious to a person of ordinary skill in the art before the effective filling date to combine Cubicasa teaching of providing updated amount of time it takes based on space and complicity for one room and 4 room, to the modified model in order to increase the accuracy of generation process with the error margin of around 5%(Cubicasa, FAQ).
Claim 46 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Adsumilli; Balineedu Chowdary (US 20190387167 A1).
As of claim 46, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 32, but it does not explicitly teach the limitations of claim 46.
While Adsumilli teaches wherein the at least one recommendation for reducing the predicted amount of time includes automated instructions to adjust one or more settings of the image acquisition device ([0035], The interface 235 can also provide a communicative interface between the image server 205 and one or more cameras 200, enabling the image server 205 to receive images and other data from the cameras 200, and providing configuration or image capture instructions to the one or more cameras 200).
Adsumilli is considered to be analogous to the claimed invention since it teaches stitching of image and image capturing instruction. Therefore, it would be obvious for a person of ordinary skill in the art before the effective filing date to integrate Adsumilli ‘s teaching of enabling the server to provide configuration or capturing instruction to cameras which reads on automatic camera setting instruction into the modified model to reduce the amount of time to generate the mapping information.
The motivation would have been to improve the performance of the stitching operation by adjusting the camera setting and the stitch engine 245 can perform one or more preprocessing operations on the images corresponding to the overlap region in order to prepare the images for stitching (Adsumilli, [0046]).
Claim 47 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Fathi; Habib (US 11106911 B1).
As of claim 47, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 32, and Bichu also teaches wherein the modified predicted amount of time is less than the predicted amount of time ([0363] the stitched result in the quarter resolution case appears sufficient depending on the application. With the reduction in number of pixels (1/16th times the number of pixels) at a quarter resolution, the total time required for image stitching was brought down from 25 s to 16.3 s).
The modified model does not explicitly teach wherein at least one additional image replaces one of the images.
While Fathi teaches wherein the at least one additional image replaces one of the images ([0063], Such identification of one or more problematic image acquisition aspects or characteristics that occurred in a first image acquisition event and the reduction or elimination of such problematic aspects in a second image acquisition event can be generated automatically by creating a new capture plan for the second (or subsequent) image acquisition event or by modification of the first capture plan to include corrections as indicated from failure analysis of the output of the first image acquisition event. In this regard, correction of the second capture plan can be provided substantially without user interaction).
Fathi is considered to be analogous to the claimed invention since it focus on Image Acquisition Planning Systems. Therefore, it would be obvious for a person of ordinary skill in the art before the effective filling date to integrate Fathi teaching of acquiring a new image to replace the problematic image on the modified model of generating a mapping information.
The motivation would have been capturing a second image to replace problematic images improve the accuracy and image acquisition process, it could be expected that the 3D information derivable from an image acquisition event be improved over time (Faith, [0062] -[0066]).
Claim 48 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Williams, Don, and Peter D. Burns. "Image Stitching: Exploring Practices, Software, and Performance." Archiving Conference. Vol. 10. Society of Imaging Science and Technology, 2013.
As of claim 48, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 32, but it does not explicitly teach the limitations of claim 48.
While William teaches , the operations further comprising, before obtaining the images, generating an initial prediction of the amount of time based at least in part on computing resources available for the automated generation process and on publicly available information about the building (page 129, “ Sources of Variability & Best Practices”, There are several sources of variability in capturing the individual tiles that can make the image stitching process inaccurate and very difficult from which to correct… Computer platform - Image stitching always involves large final images (1-3 Gbytes) as well as relatively large tiles (150-250 Mbytes). A typical number of tiles per image is between four and twelve and sometimes as large as thirty. To make the processing of such large files feasible the computer platform and associated drives need to have sufficient memory, swap disk space, and often, multi-core capability. Indeed, some stitching programs monitor the progress of the stitching process and make dynamic decisions on how to best provide a final image in a reasonable amount of time. If computer resources become scarce, certain logical, adaptive, decisions in the algorithm are made. The result is often a suboptimal stitch. This sometimes manifests itself with variable results at different execution times (e.g., day 1 vs. day 2) despite providing the software with the exact same image files and processing selections).
Williams is considered to be analogous to the claimed invention since it teaches combining images to create a complete scene. Therefore, it would be obvious to try by a person of ordinary kill in the art before the effective filling date to use Williams teaching of computational resource used for image stitching and the recommended resources (feedback) for optimal stitch on to the combined model to perform generation of mapping information.
The motivation would have been Once we have a basic understanding of the steps used by current image stitching software; we can make several suggestions that will improve results. The first is to control the lighting and relative position of the component tile images so that they are approximately related by simple translation across the object. The second is to characterize the camera taking lens distortion (e.g. barrel or pincushion), and correct each image tile prior to image stitching (Williams, conclusion)
Claim 49 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view of Williams, Don, and Peter D. Burns. "Image Stitching: Exploring Practices, Software, and Performance." Archiving Conference. Vol. 10. Society of Imaging Science and Technology, 2013 further in the view of Adsumilli; Balineedu Chowdary (US 20190387167 A1)
As of claim 49, the modified model of Li- Pham- Wang -Bichu-William teaches all the limitations of claim 48, but it does not explicitly teach the limitations of claim 49.
While Adsumilli teaches wherein the initial prediction is further based at least in part on information describing at least one of the historical performances of a user associated with the image acquisition device or a priority level associated with the user ([0075], In some embodiments, the stitch engine 245 determines a likelihood that a stitching operation will produce a visible artifact when stitching together images based on a historical performance of the stitching operation. For example, if a stitching operation has resulted in a 20% rate of producing visible artifacts for previously stitched images, the stitch engine 245 can determine that the stitching operation is 20% likely to produce a visible artifact).
Adsumilli is considered to be analogous to the claimed invention since it teaches stitching of image and image capturing instruction. Therefore, it would be obvious to try for a person of ordinary skill in the art before the effective filing date to try to use historical performance of the user for initial prediction based on Adsumilli ‘s teaching of using a historical performance to determine the like hood that a stitching operation will produce visible artifact.
The motivation would have been to improve the performance of the stitching operation by adjusting the camera setting and the stitch engine 245 can perform one or more preprocessing operations on the images corresponding to the overlap region in order to prepare the images for stitching (Adsumilli, [0046]).
Claim 51 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view Bhogal; Nikhil (US 20120293607 A1).
As of claim 51, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 32 but it does not explicitly teach the limitations of claim 51.
While Bhogal teaches determining a current status of acquiring images for the automated generation process, the current status including at least one of an amount completed or an amount of the images remaining to be acquired; and outputting the current status together with the feedback ([0072], (Step 1014). As each incoming lower resolution image portion is stitched together with the growing panoramic image preview, the resultant panoramic preview image may be sent to the device's display to provide the user with a real-time or near real-time progress indicator for the panoramic image that is currently being captured (Step 1016). In some embodiments, the growing panoramic image preview may be overlaid on the device display with the scaled preview version of image referred to in Step 1004 above ... [75], the panoramic photography process 200 described herein may provide the user with a seam less user experience including real-time progress feedback on the panoramic scene being captured, while simultaneously performing image stitching in substantially real-time--a feat previously thought to be too processing-intensive to achieve using handheld personal electronic devices).
Bhogal is considered to be analogous to the claimed invention, because it focuses on panorama image processing. Therefore, it would be obvious to try by a person of ordinary skill in the art before the effective filling date, based on Bhogal's teaching of progress indicator to show the status of the image acquiring for the automated generation process and display the status on user device into the modified model so that the feedback also outputs the progress indicator.
The motivation would have been panoramic processing module may also provide
feedback image registration information to allow the motion filter module to make more
accurate decisions regarding correlating device positional movement to overlap
amounts between successive image frames in the image stream. This feedback of information may allow more efficiently select image frames that will help image
stitching. (Bhogal, para 84).
Claim 56 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view Norio Yamamoto (JPH11213143A).
As of claim 56, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 32, but it does not explicitly teach the limitations of claim 56.
While Norio teach wherein the predicted amount of time is generated using a regression model ([0034] – [0035], FIG. 7 shows an example of actually measuring the overhead processing time calculation formula TO (x) = C × x + D in the above formula (3). FIG. 7 shows 200 dpi and 300d 0.5 times, 0.8 times, 1. times., 4 types of A4 size images at 400 dpi, 400 dpi and 600 dpi. The processing time was measured for three types of enlargement / reduction processing of five times. FIG. 7 plots the relationship between the number of pixels of the output image and the processing time at this time. By performing a regression analysis, C and D in the above equation can be obtained).
Ikeda is considered to be analogous to the claimed invention since it teaches estimating processing time of image. Therefore, it would be obvious to try for a person of ordinary skill in the art before the effective filling date to use regression model to predict amount of time based on Ikeda ‘s teaching of determining the processing time using equation for the modified model.
The motivation would have been to help estimating processing time and it perform a high-speed processing using multi-CPU system having the same processing time as the actual processing time and executes the image processing in parallel with other modules that can predict the processing time (Norio, [0050]).
Claim 57 is rejected under 35 U.S.C. 103 as being unpatentable over LI, YUGUANG (CA 3090629 A 1), in the view of Pham, Nam Thanh, Sihyun Park, and Chun-Su Park. "Fast and efficient method for large-scale aerial image stitching." IEEE Access 9 (2021), in the view of Wang; Mingyu (US 20160198088 A1), in the view of Bichu; Tanmay Nitin (US 20210082086 A1), further in the view GANI HAR (US 20210232719 A1).
As of claim 57, the modified model of Li- Pham- Wang -Bichu teaches all the limitations of claim 32, but it does not explicitly teach the limitations of claim 57.
While Gani teaches wherein the mapping information includes at least one of a three-dimensional model of an interior of the building or a virtual tour involving user-selectable inter-connections between at least some of the
Images ([0008], It is a general object of the present disclosure to provide system and method for generating and visualizing an interior design pertaining to a floor plan in a three-dimensional representation).
Gani is considered to be analogous to the claimed invention since it focuses on
interior design pertaining to a floor plan. Therefore, it would be obvious for a person of
ordinary skill in the art, before the effective filling date to integrate Gani teaching of
interior 3D model into the modified model.
The motivation would have been it would be appreciated that, creation of 3D
representation of the interior design allows a user to visualize placement of various
objects such as doors, windows, lighting, furniture, fixtures, etc. in various spaces of a
floor plan (Gani [0108]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Childs; Bradley McKay (US 11314905 B2, Date Published, 2022-04-26) this
invention is also analogous to the claimed invention since it focus on using a
computing device to allow a user to quickly and conveniently take
measurements of interior building features, and to create computerized floor
plans of such features from any location within a space, without requiring the
user to stay in a single location while taking the measurements
Dawson; Mitchell David (US 10708507 B1 Date Published 2020-07-07) this
invention is also analogous to the claimed invention since it teaches
automated operations to control acquisition of images in a defined area,
including obtaining and using data from one or more hardware sensors.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABRHAM A. TAMIRU whose telephone number is (571)272-6987. The examiner can normally be reached Monday - Friday 8:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at 571 272 4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABRHAM ALEHEGN TAMIRU/ Examiner, Art Unit 2188
/AKASH SAXENA/Primary Examiner, Art Unit 2188
