DETAILED ACTION
Response to Arguments
This office action is responsive to the communication filed on 04/02/2026. As an initial matter, the 35 USC 112/102 claim interpretation and rejections set forth in the previous office action have been withdrawn in view of Applicant's amendments and arguments.
Applicant's remaining arguments regarding the 35 USC 103 rejections with respect to the amended limitations of the claims 1, 4-10, 12-21 have been fully considered but they are not persuasive.
Applicant argues in pages 2-3 RE the amended limitations of “constructing a three-dimensional, 3D, model of contents within the storage container from the plurality of images and the corresponding plurality of angular positions, wherein the 3D model is constructed using photogrammetry and wherein the 3D model is a textured 3D mesh model” against the references individually that “Abdoo does not disclose using the angular position of the door (θ) as a variable in a photogrammetric equation to generate a 3D structural model of the items themselves.” And “While Eirinberg mentions 3D avatars in the context of augmented reality experiences (Eirinberg at [0067]), it is silent on the specific challenge of constructing a 3D model of refrigerator contents from a door-mounted camera sweeping through known angles. Eirinberg’s discussion of 3D avatars is for social media and AR experiences, not for structural reconstruction of cluttered environments using photogrammetry with known angular sweeps.”. In response, the examiner contests that one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Here Abdoo is modified by to teach the limitations as a whole under the 103 obviousness, wherein Abdoo readily teaches “constructing a three-dimensional, 3D, model of contents within the storage container from the plurality of images and the corresponding plurality of angular positions” in Figs 3-5, [0079]-[0082] successive scan data from the camera (including color cameras) corresponding to the plurality of angular positions are stitched, combined, or otherwise aggregated generating a three-dimensional representation of the interior of the refrigerating compartment. While Abdoo is silent RE, wherein the 3D model is constructed using photogrammetry and wherein the 3D model is a textured 3D mesh model, Eirinberg is relied upon to modify Abdoo teaching generating and displaying the 3D model utilizing typical photogrammetry calculation and geometric transformation with mesh generation with texture mapping in [0081]-[0084], [0115], [0047] etc, allowing the user to see the contents inside of the refrigerator [0042]. This is readily available or can be equally applied in Abdoo to effectively generate the 3D image/model from the successive scan data from the camera (including color cameras) corresponding to the plurality of angular positions utilizing typical photogrammetry calculation and geometric transformation with mesh generation with texture mapping, as readily recognized by one of ordinary skill in the art.
In addition Appicant’s argument in page 4 that “Navarrette does not disclose receiving a plurality of images from a door-mounted sensor at specific angular positions to build that model.” Is clearly not persuasive, as Navarrette is not cited for teaching the feature of “receiving a plurality of images from a door-mounted sensor at specific angular positions to build that model”. In contrast Abdoo already teaches this feature, as clearly set forth above. However please note Navarrette also teaches this feature in [0048] “sensor data from the external sensor(s) 215A and 215B and internal sensor(s) 217A and 217A and generating integration data that may be used to render the three-dimensional representation for the automobile 210A, the appliance 210B or any other object … integrating the data sets in real-time, and of rendering any augmented reality or other three-dimensional representation”, utilizing internal sensor data from typical household appliances including refrigerator [0046].
In response to applicant's argument in page 3 that “One of ordinary skill in the art would not look to a social media messaging application to solve the problem of physical occlusion in a refrigerator. The combination of Abdoo and Eirinberg would not yield the claimed invention, as neither reference teaches nor suggests using the specific angular position data of the door to construct a photogrammetric 3D model.”, the examiner's conclusion of obviousness is based upon improper hindsight reasoning, it must be recognized that any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into account only knowledge which was within the level of ordinary skill at the time the claimed invention was made, and does not include knowledge gleaned only from the applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971). Here both Abdoo and Eirinberg are directed to generate 3D models of a refrigerator contents with multiple angle/FOV photographs from a camera inside of the refrigerator with typical calculation, transformations etc, Abdoo generalized as stich, combine/aggregates [0079], Eirinberg disclosing specific details in [0081]-[0084]. The motivation to combined directly from Eirinberg [0042] which allows the user to see the contents inside of the refrigerator using the camera inside of the refrigerator, such as without opening the refrigerator door.
In response to applicant's argument in page 4 that “Navarrette is directed to an augmented reality display system, such as a car windshield or appliance door, that tracks a viewer's head to change the perspective of a virtual object overlaid on the real world (Navarrette at [0045], [0117]). The present claims, however, are directed to changing the perspective from which a pre-constructed static 3D model is rendered. Navarrette’s system is dynamic AR content, whereas Applicant’s invention is interactive navigation of a captured 3D space. The function and purpose are different. Navarrette tracks the user to make a virtual object appear anchored in real space. Applicant’s invention tracks the user to allow them to “look around” a static, previously captured environment (like looking through a window).” Indicating Navarrette is nonanalogous art, it has been held that a prior art reference must either be in the field of the inventor’s endeavor or, if not, then be reasonably pertinent to the particular problem with which the inventor was concerned, in order to be relied upon as a basis for rejection of the claimed invention. See In re Oetiker, 977 F.2d 1443, 24 USPQ2d 1443 (Fed. Cir. 1992). In this case, Navarrette is relied upon to teach the feature of “tracking a headpose of a viewer of the image display and/or further comprising changing the perspective from which the displayed image is rendered from the 3D model based on the headpose of the viewer” in dependent claim 9 in Fig 12, [0045], [0048], [0117], [0129] etc for displaying the 3D model based on users perspective/head pose, solving the problem of continuously updating the view based on users head pose changing wherein the head tracking and display update can be used on any field of endeavor, for interactive display, as understood in common knowledge. In addition Navarrette is not restricted to display on automatic cars, but displays on daily appliances like microwave and refrigerator [0047] tracking the viewers head, and to render the three-dimensional representation [0048], that incorporates movement both internally and externally to the display and takes into account the perspective of a viewer so that three-dimensional content may properly perspective shift based upon movement of the viewer or viewers (Abstract).
Therefore as clearly set forth above, the applied references clearly satisfies the claimed requirement. Hence rejection of the claims are maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-7, 10, 12-16, 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Abdoo et al (US 20180239319 A1), and further in view of Eirinberg et al (US 20230136321 A1).
RE claim 1, Abdoo teaches A computer-implemented modelling method (abstract, Fig 8) comprising:
receiving a plurality of images from an image sensor attached to a side of a hinged door facing an interior of a storage container, wherein each of the plurality of images is captured when the hinged door is at a corresponding one of a plurality of angular positions (θ) relative to the storage container (Figs 6-7, [0079]), wherein the plurality of angular positions comprise a plurality of predetermined angular positions, and the method further comprises instructing the image sensor to capture an image when an instantaneous angular position is substantially equal to one of the plurality of predetermined angular positions (Figs 6-7, [0077]-[0079], [0088]-[0089]); and
constructing a three-dimensional, 3D, model of contents within the storage container from the plurality of images and the corresponding plurality of angular positions (Figs 3-5, [0079] “the perspective of each camera of the scanning apparatus 544 will be slightly moved during each successive scan 660 of the data capturing phase 550, such that, when the data is stitched, combined, or otherwise aggregated, the resulting data represents a three-dimensional image … of the interior 572 of the refrigerating compartment 518.”, [0082]).
Abdoo is silent RE, wherein the 3D model is constructed using photogrammetry and wherein the 3D model is a textured 3D mesh model.
However Eirinberg teaches in [0081]-[0086], [0115], [0047] etc for displaying the 3D model utilizing typical photogrammetry and/or mesh generation with texture mapping, allowing the user to see the contents inside of the refrigerator [0042]. This is readily available or can be equally applied in Abdoo to effectively generate the 3D image/model from the successive scan data from the camera (including color cameras) corresponding to the plurality of angular positions utilizing typical photogrammetry calculation and geometric transformation with mesh generation with texture mapping, as readily recognized by one of ordinary skill in the art.
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include in Abdoo a system and method wherein the 3D model is constructed using photogrammetry and wherein the 3D model is a 3D mesh model, optionally a textured 3D mesh model, as set forth above applying Eirinberg, in order to display the 3D model and thereby increasing system effectiveness and user experience.
RE claim 4, Abdoo teaches wherein every successive two of the plurality of predetermined angular positions are a constant angle apart and the constant angle is 5 degrees (Figs 6-7, [0078]-[0079] wherein the predetermined rotational position of each door relative to the open and/or closed position anticipates the predefined constant angles that can be set as 5 degrees as an obvious design/system choice).
RE claim 5, Abdoo teaches further comprising recording the instantaneous angular position at the time when an image is received from the image sensor as the angular position corresponding to the received image (Figs 6-7, [0079]).
RE claim 6, Abdoo teaches further comprising moving the image sensor relative to the hinged door as the angular position of the hinged door relative to the storage container changes and/or wherein the image sensor is moved relative to the hinged door in a direction parallel to the rotational axis of the hinged door as the angular position of the hinged door relative to the storage container changes (Figs 6-7, [0077]- [0079]).
RE claim 7, Abdoo teaches further comprising updating the 3D model each time a plurality of new images captured at a corresponding plurality of angular positions is received as the hinged door is being moved away from and/or towards a closed position (Figs 6-7, [0061], [0077]-[0079]).
Claims 10, 12-16 recite limitations similar in scope with limitations of claims 1, 3-7 and therefore rejected under the same rationale. In addition Abdoo teaches A modelling system configured to model contents within a storage container, the system comprising: an image receiver and a model constructor (Fig 1, [0056]).
RE claim 19, Abdoo teaches A storage container comprising a hinged door, and the system of claim 10 (Fig 1, [0056]).
RE claim 20, Abdoo teaches wherein the image sensor is moved relative to the hinged door in a direction parallel to the rotational axis of the hinged door as the angular position of the hinged door relative to the storage container changes (Figs 6-7, [0077]- [0079]).
RE claim 21, Abdoo teaches wherein the storage container is a refrigerator or a refrigerator compartment (Figs 1,4-5 [0056]).
Claims 8-9, 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Abdoo as modified by Eirinberg, and further in view of Navarrette et al (US 20200151943 A1).
RE claim 8, Abdoo as modified by Eirinberg teaches further comprising displaying an image on an image display and/or wherein the image display is mounted on an exterior-facing side of the hinged door (Abdoo [0090], Eirinberg [0042], [0081]).
Abdoo as modified by Eirinberg is silent RE the image rendered from the 3D model. However Navarrette teaches in Fig 12, [0022], [0048] etc for displaying the 3D model wherein the image display is mounted on an exterior-facing side of the hinged door (Figs 10, 12-13, [0122]). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include in Abdoo a system and method wherein the image rendered from the 3D model, as suggested by Navarrette, in order to effectively display the 3D model on the 2D screen and thereby increasing system effectiveness and user experience.
RE claim 9, Abdoo as modified by Eirinberg is silent RE further comprising tracking a headpose of a viewer of the image display and/or further comprising changing the perspective from which the displayed image is rendered from the 3D model based on the headpose of the viewer.
However Navarrette teaches in Fig 12, [0045], [0048], [0117], [0129] etc for displaying the 3D model based on users perspective/head pose.
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include in Abdoo as modified by Eirinberg a system and method of tracking a headpose of a viewer of the image display and/or further comprising changing the perspective from which the displayed image is rendered from the 3D model based on the headpose of the viewer, as suggested by Navarrette, or displaying the 3D model based on users perspective/head pose and thereby increasing system effectiveness and user experience.
Claims 17-18 recite limitations similar in scope with limitations of claims 8-9 and therefore rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. (See attached 892).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SULTANA MARCIA ZALALEE whose telephone number is (571)270-1411. The examiner can normally be reached Monday- Friday 8:00am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Sultana M Zalalee/Primary Examiner, Art Unit 2614
Prosecution Insights