DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 18-19, 21, 25, and 30-32 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa et al. (US 20240005619 A1, hereinafter “Yoshizawa”).
Regarding claim 1, Yoshizawa discloses An information processing apparatus comprising: (Abstract, “An augmented reality processing system transmits information regarding an augmented reality object(ARO) arranged in a virtual space on the basis of a world-coordinate-system from an information display device to an information processing device”). Note that: the system is an information processing apparatus.
an acquisition circuit configured to acquire target space data acquired as measurement is performed inside a target space from a plurality of positions; and (para. [0071], “a position information calculation unit 1111 which calculates the position (coordinates) of the HMD 100 in the real world on the basis of a sensor output from the position sensor 151 to generate position information, an object information acqms1t10n unit 1112 which acquires AR object information related to the position (coordinates) of the HMD 100 in the real world, an orientation information calculation unit 1113 which calculates the orientation related to the direction that the HMD 100 points in the real world, on the basis of a sensor output from the sensor group 160 such as the gyro sensor 161 and the geomagnetic sensor 162 to generate orientation information, an object information display control unit 1114 which performs display control of an AR object configured on the basis of the AR object information and transmission control of the AR object information to the linked tablet terminal 200, and a temporary storage area 1199 which temporarily stores various information created or acquired by each of the above-described function units.”). Note that: (1) the real world in which the HMD device is located is a target space; (2) an object information acquisition unit 1112 is an acquisition circuit, and acquire the target space data as the output of the sensor group and AR object information related to the position of the HMD160 from a set of positions when the HMD moves around in the target space; and (3) an orientation information calculation unit 1113 performs the calculation or measurement.
a processor configured to process the acquired data, the processor being configured to (para. [0070], “the processor 101 execute the expanded basic operation program 1001 and application program 1002 to thereby realize their functions”).
recognize, from the acquired target space data, an attribute and a position of an object in the target space, the object affecting an environment in the target space, and (para. [0101], “The AR object 410 is arranged in the world coordinate system virtual space. The world coordinate system virtual space is a virtual space which shares position information with the real world”; para. [0107], “In the AR object 410, in advance, (X (obj), Y (obj)) is set as position (coordinate) information of a reference position 410P, and D (obj) is set as orientation information 410D”; para. [0190], “The AR object 410 that reflects the changes in shape, display color, etc. may be determined by AR object identification information”; para. [0212], “an added AR object comprised of a comment is added to a reality object comprised of an image of a real object existing in a real space.”). Note that: (1) an AR object as a comments affects a reality object of the environment in the target space; (2) the AR object’s coordinates are position information; and (3) the D (obj) is set as orientation information as an attribute as well as shape and display color, etc.
output the attribute and the position of the object in the target space in association with the target space data. (para. [0107], “When the position (coordinate) information (X (obj), Y (obj)) of the reference position 410P falls within the field of view range of the user Ul via the display 131 of the HMD 100, the AR object 410 is displayed at the position on the display 131 corresponding to the position (coordinate) information (X (obj), Y (obj)) and in the direction corresponding to the orientation information D (obj)”). Note that: the AR object 410 is displayed or output onto the display 131 based on the position and attribute information.
Before the effective filing date of the claimed invention, it would have been obvious to learn and use the teachings on the space data acquisition, object position and attribute determination, and output and display of AR object on a display, as taught by Yoshizawa. Although outputting the attribute and the position of the object are not explicitly described by Yoshizawa, it is obvious for one having ordinary skill in the art to understand performing outputting the attribute and the position of the object to a storage to device so that the object can then be displayed or output on a display. The motivation would have been “When the position (coordinate) information (X (obj), Y (obj)) of the reference position 410P falls within the field of view range of the user Ul via the display 131 of the HMD 100, the AR object 410 is displayed at the position on the display 131 corresponding to the position (coordinate) information (X (obj), Y (obj)) and in the direction corresponding to the orientation information D (obj)” (para. [0107]). The suggestion for doing so would allow to output the attribute and the position of the object in the target space in association with the target space data. Therefore, it would have been obvious to use the teachings by Yoshozawa.
Regarding claim 2, Yoshizawa discloses The information processing apparatus according to claim 1, wherein based on the target space data acquired by the acquisition circuit and the attribute and the position of the object in the target space, an augmented reality image of the target space in which the attribute and the position of the object in the target space, the object affecting the environment in the target space, are reflected is created. (para. [0132], “The AR object fixing processing in the tablet terminal 200 will be described with reference to FIGS. 13B and 14. FIG. 13B is a diagram showing an image displayed on the touch screen 290 of the tablet terminal 200 by the AR object fixing processing”). Note that: the AR image with the AR object displayed on a screen indicates that the AR image is generated or created reflecting the attribute and the position of the object in the target space while the object affecting the environment in the target space.
Regarding claim 3, Yoshizawa discloses The information processing apparatus according to claim 1, wherein based on the target space data acquired by the acquisition circuit and the attribute and the position of the object in the target space, a three-dimensional space model of the target space in which the attribute and the position of the object in the target space, the object affecting the environment in the target space, are reflected is created. (para. [0230], “The object information display control unit 1114 generates a 3D model (reality object information) regarding the reality object 400 selected in the processing of S605, on the basis of the distance information about the reality object 400 acquired in the processing of S606 and the reality object image acquired by the out-camera 133 (S608). Incidentally, it is sufficient if the 3D model generated by this processing can be recognized as a position reference when adding a provisional added AR object such as a comment in the provisional object edit processing performed on the tablet terminal 200, and it is not necessary to accurately reproduce the shape of the reality object 400. Further, the reality object information includes the position information and orientation information regarding the reality object calculated in the processing of S607”). Note that: the 3D model(s) of object(s) with corresponding attributes and position information of the target space constitute the 3D target space model.
Regarding claim 18, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the processor is configured to accept editing of the attribute and the position of the object in the target space to change a three-dimensional space model. (para. [0006], “accepts an edit operation for the displayed provisional augmented reality object, and transmits information about the edit operation for the provisional augmented reality object to the information display device”; para. [0165], “The processing of editing the provisional AR object 430 fixed to the display 231 of the tablet terminal 200 by the AR object fixing processing will be described with reference to FIGS. 16A, 16B, and 17. In the edit processing of the provisional AR object 430, it is possible to perform changing of the shape, display color, and the like of the provisional AR object 430, addition of a comment and the like associated with the provisional AR object 430, etc.”; para. [0169], “it is possible to change the display position of the provisional AR object 430 in the display 231. [0170]”). Note that: (1) the attributes (shape and color) and the position of an object can be changed in the target space; and (2) the edit operation is accepted.
Regarding claim 19, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the processor is configured to accept addition of an object in the target space to change a three-dimensional space model. (para. [0006], “accepts an edit operation for the displayed provisional augmented reality object, and transmits information about the edit operation for the provisional augmented reality object to the information display device”; para. [0177], “the object information display control unit 2113 confirms whether or not there is changing the shape, display color, etc. of the provisional AR object, or adding the provisional added AR object (S402).”). Note that: an object can be added as an edit operation, and the edit operation is accepted.
Claim 30 reciting “An information processing system comprising” is corresponding to the apparatus of claim 1. Therefore, claim 30 is rejected for the same rationale for claim 1.
In addition, Yoshizawa discloses An information processing system comprising (Abstract, “An augmented reality processing system transmits information regarding an augmented reality object(ARO) arranged in a virtual space on the basis of a world-coordinate-system from an information display device to an information processing device”).
Regarding claim 21, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the processor is configured to use self-position estimation processing or visual line direction estimation processing to identify the target space and the position of the object in the target space. (Yoshizawa, para. [0215], “When the user Ul visually recognizes the reality object 400, the object information display control unit 1114 calculates to which position on the display 131 the line-of-sight position 460 of the user Ul corresponds, on the basis of the eye image of the user Ul acquired by the in-camera 134”; para. [0224], “Based on the position information signal acquired from the position information transmitting device 320 via the position sensor 151, the position information calculation unit 1111 calculates position (coordinate) information (X(hmd2), Y (hmd2)) of the HMD 100 in the real world at the time of executing the reality object fixing processing (S602).”). Note that: the process using visual line direction (light-of-sight) estimated or calculated with position information of the HMD vias position sensor 151 can visually recognize or identify the target space and the position of the object in the target space.
Regarding claim 25, aaa discloses The information processing apparatus according to claim 1, wherein the target space data includes … and the processor is configured to …
RGB data and time data, (Yoshizawa, para. [0215], “When the user Ul visually recognizes the reality object 400, the object information display control unit 1114 calculates to which position on the display 131 the line-of-sight position 460 of the user Ul corresponds, on the basis of the eye image of the user Ul acquired by the in-camera 134.”). Note that: (1) the eye image for the target space can be acquired by the in-camera of the HMD with the timestamps indicating the acquisition time of the image; and (2) the conventional in-camera of the HMD is usually a RGB camera to acquire the RGB images.
identify a utilization scene of the target space based on the RGB data and the time data. Note that: it is well known that the orientation of the sun can be obtained base on the measurement of a shadow direction and a shadow length of a length-defined post vertical to a horizontal plane, which was developed by ancient people. It is obvious to one having ordinary skill that: a) the corresponding RGB images are acquired when the user views the shadow of the length-defined post; b) the processor can obtain or determine the orientation information using the same method developed by ancient people by measuring a shadow direction and a shadow length; and c) when the user views the shadow of the length-defined post, the processor identifies a utilization scene of the target space for determining the orientation information.
Claim 31 reciting “An information processing method comprising:” is corresponding to the apparatus of claim 1. Therefore, claim 31 is rejected for the same rationale for claim 1.
In addition, Yoshizawa discloses An information processing system comprising (Title, “AUGMENTED REALITY PROCESSING SYSTEM, INFORMATION DISPLAY DEVICE, AND AUGMENTED REALITY PROCESSING METHOD”).
Claim 32 reciting “A non transitory computer readable medium encoded with a computer program that causes a computer to execute a process comprising:” is corresponding to the apparatus of claim 1. Therefore, claim 32 is rejected for the same rationale for claim 1.
In addition, Yoshizawa discloses A non transitory computer readable medium encoded with a computer program that causes a computer to execute a process comprising: (paras. [0048]-[0049], “The HMD 100 includes: a processor 101 (corresponding to a first processor) using a CPU an MCU, or the like; a RAM 103; a storage 110 … The storage 110 is configured using a non-volatile storage medium such as a Flash ROM, an EEPROM, an SSD, an HDD, or the like”; FIG. 4: “STORAGE 110” comprises “BASIC OPERATION PROGRAM 1001” and “APPLICATION 1002”).
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Pinmas et al. (Towards Digital Twin Data Center Using Building Information Modeling and Real-Time Data Sensing, 2022 37th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC) DOI: 10.1109/ITC-CSCC55581.2022.9895024, hereinafter “Pinmas”) .
Regarding claim 4, Yoshizawa discloses The information processing apparatus according to claim 3, wherein the three-dimensional space model
However, Yoshizawa fails to disclose, but in the same art of computer graphics, Pinmas discloses is usable to predict the environment in the target space. (Pinmas, Abstract, “This paper presents an application of digital twin modeling for a data center in an academic institute. The building information modeling (BIM) has been applied to represent the data center. The sensing data (i.e., temperature, humidity, energy consumption, and computing resource utilization) in the data center are collected using the network of IoT”; page 909, col. left, para. 3, “These sensing data are visualized together with the data center model in 3D. The data center managers can closely monitor the data center readily. Then, the long short-term memory (LSTM) technique is applied to predict the future data center room temperature”). Note that: (1) the data center model in 3D as a digital twin is the three-dimensional space model; and (2) the 3D space model can be used to monitor the data center, and can predict the future data center room temperature with LSTM.
Yoshizawa and Pinmas are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply using digital twin to monitor a data center and predict the environment changes, as taught by Pinmas into Yoshizawa. The motivation would have been “the long short-term memory (LSTM) technique is applied to predict the future room temperature. The real-time data sensing of the data center facilitates the data center operator to monitor incidents in the data center” (Pinmas, Abstract). The suggestion for doing so would allow to monitor and predict the environment in the target space. Therefore, it would have been obvious to combine Yoshizawa and Pinmas.
Regarding claim 5, Yoshizawa in view of Pinmas discloses The information processing apparatus according to claim 1, wherein the object in the target space includes at least one of
an object affecting an airflow in the target space, an object affecting a thermal environment in the target space, (Pinmas, page 910, col. right, para. 4, “Fig. 3 shows an example of the temperature heatmap of the 21 server racks in the data center”; page 910, Fig. 3: “
PNG
media_image1.png
352
528
media_image1.png
Greyscale
”). Note that: (1) the server racks are can be regarded as objects affecting an airflow in the target space as a common sense since the running server racks heat up the airflows in a data center; and (2) the heat map shows that the server racks as objects affect a thermal environment in the target space.
an object affecting air quality in the target space, and an object controlling the environment in the target space (Pinmas, page 909, col. left, para. 2, “In a data center, there are several objects that should be monitored, such as the equipment energy consumption, the room temperature, humidity, and computing resource utilization”). Note that: (1) the objects of equipment energy consumption, the room temperature, and humidity are the objects that affect the room air quality in terms of room temperature, and humidity; and (2) the equipment energy consumption is an object controlling the energy dissipated in the data center and controlling the environment.
The motivation to combine Yoshizawa and Pinmas given in claim 4 is incorporated here.
Claims 6 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Busch (Characterization of Energy Processes in Buildings, 2022 37th International Technical Conference on Circuits/Systems, Fundamentals of Building Energy Dynamics, ISBN (Online): 9780262275903, MIT Press eBook, Publication Date: 1996-01-01, hereinafter “Busch”) .
Regarding claim 6, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the object in the target space includes
However, Yoshizawa fails to disclose, but in the same art of environment and space, Busch discloses
at least one of a fixture (Busch, page 178, para. 2, “The internal lighting load imposed on the HVAC system is directly related to the amount of power required by a lighting system. The efficacy of a lamp or lighting fixture”), a fitting (page 156, para. 1, “windows also transfer heat by radiation and conduction. To calculate the thermal energy performance of a building, one must know the heat transfer characteristics of the windows as a function of the environmental variables such as temperature and wind speed”). Note that: a window is a fitting according to Applicant’s definition in the specification , a person, an industrial machine, (page 122, para. 2, “Indoor-generated pollutants from human activities (C02, hydrocarbons, particulates, copy machines chemicals, aerosol sprays) and from building contents or materials (radon, formaldehyde, asbestos)”). Note that: human activities indicate persons involved in the space, and copy machines indicate industrial machine involved in the space a plant, (page 131, para. 2, “For plant equipment (furnaces, boilers, and chillers), efficiencies express the ability of that equipment to convert electrical or fuel energy into space-conditioning energy”). Note that: plant equipment indicates a plant can be an object. an air conditioner, and a ventilation device. (page 170, para. 2, “newer buildings are using VAV systems for air-conditioning and ventilation.”). Note that: a VAV system can provide an air conditioner and a ventilation device. The target space can include all the mentioned objects above.
Yoshizawa and Busch are in the same field of endeavor, namely environment and space. Before the effective filing date of the claimed invention, it would have been obvious to apply obtaining a list of objects involved in a target space and understanding them to Improve the performance of the components for more efficient use of energy, as taught by Busch into Yoshizawa. The motivation would have been “1. Understanding the mechanisms of heat transfer and storage for a component; 2. Providing information to develop mathematical models of the component; 3. Creating test, installation, and energy performance standards; and 4. Improving the performance of the components for more efficient use of energy” (Busch, page 114, para. 1). The suggestion for doing so would allow to improve the performance of the components for more efficient use of energy in a space. Therefore, it would have been obvious to combine Yoshizawa and Busch.
Regarding claim 12, Yoshizawa in view of Busch discloses The information processing apparatus according to claim 1, wherein the attribute of the object in the target space includes at least one of … a shape (Yoshizawa, para. [0165], “In the edit processing of the provisional AR object 430, it is possible to perform changing of the shape”). Note that: the shape is an attribute of an object for changing.
a dimension and a shape of the object (Busch, page 158, para.3, “National Bureau of Standards studies involve simulation of the thermal and lighting characteristics of windows to determine the optimal (for energy consumption) combination of characteristics in terms of window size, heat transfer, solar shading, and orientation, for different thermostat setback strategies in commercial buildings”). Note that: the window as an object (a fitting) in the space has the window size. and sensible heat, latent heat (Busch, page 177, “Typical sensible and latent heat releases are given, as a function of activity level, in chapter 26 of the ASHRAE 1989 Handbook of Fundamentals (ASHRAE 1989a)”). Note that: sensible heat and latent heat are a function of an activity level. , an air quantity (Busch, page 172, para. 1, “if low ventilations rates (less than 0.5 air changes per hour) are adequate for suitable indoor air quality, then heat exchangers in low infiltration houses are not cost-effective except in the coldest climates and only when high-performance heat exchangers are used”). Note that: air quality is affected by ventilations rates, a direction of wind, and a wind speed in the target space. (page 175, para. 2, “Although it seems that wind direction should influence the infiltration rate, studies by Reeves, McBride, and Sepsy (1979) found that physical models including directional effects failed to account for observed flow rates and that airflow (particularly infiltration) occurs in a more general pattern, primarily dependent on wind speed and air density”). Note that: wind direction and wind speed are disclosed here for model study.
The motivation of combine Yoshizawa and Busch given in claim 6 is incorporated here.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Huo et al. (A Study of Simulation of the Urban Space 3D Temperature Field at a Community Scale Based on High-Resolution Remote Sensing and CFD, Remote Sens. 2022, 14, 3174. https://doi.org/10.3390/rs14133174, hereinafter “Huo”) .
Regarding claim 13, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the processor is configured to create a three-dimensional space model including
However, Yoshizawa fails to disclose, but in the same art of computer graphics, Huo discloses
thermal boundary conditions in accordance with the attribute of the object in the target space. (Huo, page 6, para. 1, “In this study, the dataset, including high-resolution, remotely sensed images and meteorological data, were preprocessed and used to build a three-dimensional (3D) geometric model. The parameters of the materials and the conditions of boundary were set”; page 26, para. 5, “These parameters include: (1) the underlying surface data of different thermophysical properties that cause differences in solar radiation heat; (2) meteorological boundary condition data, such as the temperature and wind speed in a study area; and (3) high-quality grid data for the study area through numerical computing technology.”; page 23, para. 2, “We simplified the actual building as a building wall, introduced it into the CFD, and set it as concrete of a certain thickness to participate in the simulation calculation, ignoring the influence of the actual difference in the building surface on the results” ). Note that: (1) the meteorological boundary condition is a thermal boundary condition and meteorological data that are used to build or create a 3D space model; (2) the walls are the objects in the target space while the sickness is the attribute of the walls; and (3) the thermal boundary conditions can be set accordingly with regards to the wall thickness.
Yoshizawa and Huo are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply setting thermal boundary conditions according to an object’s attributes, as taught by Huo into Yoshizawa. The motivation would have been “This study used high-resolution remote-sensing technology and CFD models to carry out a simulation study of a three-dimensional (3D) USTE for daytime and nighttime at a block scale.” (Huo, page 1, Abstract). The suggestion for doing so would allow to set the thermal boundary conditions according to the objects’ attributes when building a 3D space model. Therefore, it would have been obvious to combine Yoshizawa and Huo.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Huo and Borrmann et al. (Thermal 3D mapping of building façades. In Intelligent Autonomous Systems 12: Volume 1 Proceedings of the 12th International Conference IAS-12, Held June 26–29, 2012, Jeju Island, Korea; Springer: Berlin/Heidelberg, Germany, 2013; pp. 173–182, hereinafter “Borrmann”) .
Regarding claim 16, Yoshizawa in view of Huo discloses The information processing apparatus according to claim l, wherein the processor is configured to
… including a temperature of the object as a thermal boundary condition. (Huo, page 6, para. 1, “In this study, the dataset, including high-resolution, remotely sensed images and meteorological data, were preprocessed and used to build a three-dimensional (3D) geometric model. The parameters of the materials and the conditions of boundary were set”; page 26, para. 5, “These parameters include: (1) the underlying surface data of different thermophysical properties that cause differences in solar radiation heat; (2) meteorological boundary condition data, such as the temperature and wind speed in a study area; and (3) high-quality grid data for the study area through numerical computing technology.”; page 1, para. 2, “remote sensing usually obtains surface temperature information” ). Note that: (1) the meteorological boundary condition is a thermal boundary condition; and (2) the object’s surface temperature by remote sensing can be used as a parameter to set the thermal boundary condition.
The motivation to combine Yoshizawa and Huo given in claim 13 is incorporated here.
However, Yoshizawa in view of Huo fails to disclose, but in the same art of computer graphics, Borrmann discloses
acquire thermography data of the target space, and create, based on the acquired thermography data, a three-dimensional space model including a temperature of the object as a thermal boundary condition. (Borrmann, Abstract, “A robot equiped with a 3D laser scanner, a thermal camera, and a color camera constitutes the basis for our approach. The data from all three sensors and from different locations are joined into one high-precise 3D model that shows the heat distribution”). Note that: (1) the thermography data are acquired from a thermal camera; and (2) based on the thermography data and the data from other two sensors, a three-dimensional space model that shows the heat distribution for objects.
Yoshizawa in view of Huo, and Borrmann, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply acquiring thermography data of the target space, and creating based on the acquired thermography data a three-dimensional space model, as taught by Borrmann into Yoshizawa in view of Huo. The motivation would have been “To eliminate heat and air conditioning losses in buildings and factories heat and air leaks need to be localized and identified.” (Borrmann, Abstract). The suggestion for doing so would allow to acquire thermography data and create a three-dimensional space model based on the acquired thermography data. Therefore, it would have been obvious to combine Yoshizawa, Huo, and Borrmann.
Claims 20 and 29 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Cao et al. (Multi-sensor spatial augmented reality for visualizing the invisible thermal information of 3D objects, Optics and Lasers in Engineering, Elsevier Ltd., hereinafter “Cao”) .
Regarding claim 20, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the acquisition circuit is configured to
However, Yoshizawa fails to discloses, but in the same art of computer graphics, Cao discloses
acquire the target space data acquired when a measuring device that measures a physical quantity related to a shape inside the target space is moved or rotated, and
the measuring device includes an RGB sensor, a depth sensor, an infrared projector, an infrared camera, a gripper, and a terminal holder. (Cao, Abstract, “we build a multi-sensor system consisting of a long-wave infrared camera, a RGBD camera, and a digital projector for multimodal data acquisition, real-time 3D thermographic reconstruction, and projector-based spatial augmented reality”; page 106634, col. right, Fig. 7: “Infrared camera”, “Digital Projector”, “An illustration of the proposed hybrid method to handle object self- occlusion. It combines the benefits of model-based ray casting and frame-based image warping”, and “
PNG
media_image2.png
508
598
media_image2.png
Greyscale
”; Fig 2: “
PNG
media_image3.png
316
588
media_image3.png
Greyscale
”; page 106634, col. left, para. “We move the portable multi-sensor system around the sample to ac- quire thermal, RGB, and depth images and perform real-time 3D thermographic reconstruction.”; page 106634, col. left, para. 5, “We utilized the built multi-sensor system to acquire thermal, RGB, and depth images at different viewpoints. 2D thermal and 3D geometrical data are spatially registered and then integrated into a truncated signed distance function model for real-time 3D thermographic reconstruction. Finally, the proposed hybrid method harvesting the benefits of model-based ray casting and frame-based image warping is applied to generate high-quality projected images with complete view coverage and sharp thermal details for SAR via a digital projector.”). Note that: (1) the multi-sensor system as a measuring device moves in the space to acquire the target space data and measure the object depth data (a physical quantity) using the depth images related to a shape (surface) inside the target space; and (2) the measuring device include RGBD camera (RGB sensor and depth sensor for RGB and depth images), a digital infrared digital projector, an infrared camera, a camera holder in Fig. 2 above as a gripper, and a system frame in Fig. 2 above as a terminal holder.
Yoshizawa and Cao are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply acquiring a target space data with a multi-sensor system to measure the depth data and temperature data related to a surface inside the space, as taught by Cao into Yoshizawa. The motivation would have been “We utilized the built multi-sensor system to acquire thermal, RGB, and depth images at different viewpoints. 2D thermal and 3D geometrical data are spatially registered and then integrated into a truncated signed distance function model for real-time 3D thermographic reconstruction” (Cao, page 106634, col. left, para. 5). The suggestion for doing so would allow to acquire target space data with a multi-sensor system to measure the depth data and temperature data related to a surface inside the space. Therefore, it would have been obvious to combine Yoshizawa and Cao.
Regarding claim 29, Yoshizawa in view of Cao discloses An information processing system including the information processing apparatus according to claim 1, the information processing system further comprising:
a measuring device including an RGB sensor, a depth sensor, an infrared projector, an infrared camera, a gripper, and a terminal holder, the measuring device being configured to measure a physical quantity related to a shape inside the target space, the information processing apparatus being configured to acquire the target space data from the measuring device. (Cao, Abstract, “we build a multi-sensor system consisting of a long-wave infrared camera, a RGBD camera, and a digital projector for multimodal data acquisition, real-time 3D thermographic reconstruction, and projector-based spatial augmented reality”; page 106634, col. right, Fig. 7: “Infrared camera”, “Digital Projector”, “An illustration of the proposed hybrid method to handle object self- occlusion. It combines the benefits of model-based ray casting and frame-based image warping”, and “
PNG
media_image2.png
508
598
media_image2.png
Greyscale
”; Fig 2: “
PNG
media_image3.png
316
588
media_image3.png
Greyscale
”; page 106634, col. left, para. “We move the portable multi-sensor system around the sample to ac- quire thermal, RGB, and depth images and perform real-time 3D thermographic reconstruction.”; page 106634, col. left, para. 5, “We utilized the built multi-sensor system to acquire thermal, RGB, and depth images at different viewpoints. 2D thermal and 3D geometrical data are spatially registered and then integrated into a truncated signed distance function model for real-time 3D thermographic reconstruction. Finally, the proposed hybrid method harvesting the benefits of model-based ray casting and frame-based image warping is applied to generate high-quality projected images with complete view coverage and sharp thermal details for SAR via a digital projector.”). Note that: (1) the multi-sensor system as a measuring device moves to acquire the target space data and measure the object depth data (a physical quantity) using the depth images related to a shape (surface) inside the target space; (2) the measuring device include RGBD camera (RGB sensor and depth sensor for RGB and depth images), a digital infrared digital projector, an infrared camera, a camera holder in Fig. 2 above as a gripper, and a system frame in Fig. 2 above as a terminal holder; and (3) when the measuring device moves in the target space, the information processing apparatus acquired the target space data.
The motivation to combine Yoshizawa and Cao given in claim 20 is incorporated.
Claims 22-24 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Noriaki et al. (JP 2016027480 A, hereinafter “Noriaki”). A machine-translated English version for Noriaki is attached.
Regarding claim 22, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the acquisition circuit is configured to … and the processor is configured to …
However, Yoshizawa fails to disclose, but in the same art of computer graphics, Noriaki discloses
acquire orientation information related to an orientation in the target space, and the processor is configured to create a three-dimensional space model including the orientation information acquired by the acquisition circuit or information based on the orientation information. (Noriaki, page 10, lines 11-12, “a three-dimensional model is set based on the solar orientation obtained from a sensor or the like”). Note that: solar orientation is obtained or acquired from a sensor; and (2) a 3D space model can be set or created based on the orientation information.
Yoshizawa and Noriaki are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply acquiring orientation information and build a 3D space model, as taught by Noriaki into Yoshizawa. The motivation would have been “a three-dimensional model is set based on the solar orientation obtained from a sensor or the like.” (Noriaki, page 10 lines 11-12). The suggestion for doing so would allow to obtain a target space data with a multi-sensor system to acquire orientation information and create a 3D space. Therefore, it would have been obvious to combine Yoshizawa and Noriaki.
Regarding claim 23, Yoshizawa in view of Noriaki discloses The information processing apparatus according to claim 22, wherein the orientation information is information based on a latitude and a longitude of and a time in the target space and a position of a celestial body or information based on the latitude and the longitude of and the time in the target space and a direction of a shadow in a room and a length of the shadow. (Noriaki, page 10, line 5-15, “it is possible to acquire the direction of the sun in the month and time set by the user, set the light source, and display a more realistic landscape. The sun direction can be acquired from the sun direction table 710. Get azimuth and altitude by month and time. Note that the sun direction may be obtained by acquiring the current sun direction using a sensor or the like installed in the real world and reflecting it in the three-dimensional model … a three-dimensional model is set based on the solar orientation obtained from a sensor or the like. As yet another embodiment, there is a method of calculating the azimuth of the sun from the latitude and longitude of the place where the apartment is scheduled to be built and the month and time zone input on the MR system setting screen 600. That is, the method for calculating the direction of the sun is not limited to these, and there are various methods.”). Note that: (1) the orientation information (the azimuth of the sun) of sun can be calculated by a latitude and a longitude of and a time (month and time zone) in the target space; and (2) it is well known that the orientation of the sun can be obtained base on the measurement of a shadow direction and the shadow length of a length-defined post vertical to a horizontal plane, which was developed by ancient people.
The motivation to combine Yoshizawa and Noriaki given in claim 22 is incorporated here.
Regarding claim 24, Yoshizawa in view of Noriaki discloses The information processing apparatus according to claim l, wherein the target space data includes … and the processor is configured to …
RGB data and time data, (Yoshizawa, para. [0215], “When the user Ul visually recognizes the reality object 400, the object information display control unit 1114 calculates to which position on the display 131 the line-of-sight position 460 of the user Ul corresponds, on the basis of the eye image of the user Ul acquired by the in-camera 134.”). Note that: (1) the eye image for the target space can be acquired by the in-camera of the HMD with the timestamps indicating the acquisition time of the image; and (2) the conventional in-camera of the HMD is usually a RGB camera.
identify an orientation in the target space based on the RGB data and the time data, and Note that: it is well known that the orientation of the sun can be obtained base on the measurement of a shadow direction and a shadow length of a length-defined post vertical to a horizontal plane, which was developed by ancient people. It is obvious to one having ordinary skill that: a) the corresponding RGB images are acquired when the user views the shadow of the length-defined post; and b) the processor can obtain or determine the orientation information using the same method developed by ancient people by measuring a shadow direction and a shadow length.
create the three-dimensional space model including orientation information indicating the identified orientation or information based on the orientation information. (Noriaki, page 10, lines 11-12, “a three-dimensional model is set based on the solar orientation obtained from a sensor or the like”). Note that: solar orientation is obtained or acquired from a sensor; and a 3D space model can be set or created based on the orientation information.
The motivation to combine Yoshizawa and Noriaki given in claim 22 is incorporated here.
Claims 26-27 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Chen et al. (US 20150084755 A1, hereinafter “Chen”).
Regarding claim 26, Yoshizawa discloses The information processing apparatus according to claim 1, wherein the processor is configured to
However, Yoshizawa fails to disclose, but in the same art of computer graphics, Chen discloses discriminate whether or not the target space data includes the object in the target space in an overlapping manner. (Chen, para. [0023], “five cameras 110-114 may be used, which may be located, for example, in the rear, the front and the side mirrors. Because of the camera locations in the vehicle 101, the images captured might have an overlap with each other, which may be eliminated by the image combiner 116. Overlapping areas of the images may be detected and deleted enabling the image combiner 116 to generate an image of the surroundings of the vehicle 101 with no overlap”). Note that: (1) to cover the whole scene, 5 cameras capture the images for the target space while the images can be regarded as the target space data; (2) the captured images may have the common overlap areas with the same objects shown or placed; and (3) since the overlap may be detected, the processor can discriminate the images (the target space data) includes the overlap with common objects in the scene from different cameras.
Yoshizawa and Chen are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply obtaining the target space data and images and discriminating the target space data including the overlap with the common objects in the scene from different cameras and deleting the overlap, as taught by Chen into Yoshizawa. The motivation would have been “Overlapping areas of the images may be detected and deleted enabling the image combiner 116 to generate an image of the surroundings of the vehicle 101 with no overlap” (Chen, para. [0023]). The suggestion for doing so would allow to detect if the target space data include the objects in the scene in an overlapping manner and delete the overlap. Therefore, it would have been obvious to combine Yoshizawa and Chen.
Regarding claim 27, Yoshizawa in view of Chen discloses The information processing apparatus according to claim 1, wherein the processor is configured to exclude, when the object included in the target space is recognized in an overlapping manner, the overlapping object. (Chen, para. [0023], “five cameras 110-114 may be used, which may be located, for example, in the rear, the front and the side mirrors. Because of the camera locations in the vehicle 101, the images captured might have an overlap with each other, which may be eliminated by the image combiner 116. Overlapping areas of the images may be detected and deleted enabling the image combiner 116 to generate an image of the surroundings of the vehicle 101 with no overlap”). Note that: (1) to cover the whole scene, 5 cameras capture the images for the target space while the images can be regarded as the target space data; (2) the captured images may have the common overlap with the same objects shown or placed and (3) since the overlay may be detected, the processor can delete the overlaying image areas in which the common objects in the scene from different cameras are positioned, resulting in excluding the common objects in the scene.
The motivation to combine Yoshizawa and Noriaki given in claim 26 is incorporated here.
Claim 28 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshizawa in view of Becerra et al. (US 10686966 B1, hereinafter “Becerra”).
Regarding claim 28, Yoshizawa discloses The information processing apparatus according to claim 1, wherein
an attribute and a position of (para. [0101], “The AR object 410 is arranged in the world coordinate system virtual space. The world coordinate system virtual space is a virtual space which shares position information with the real world”; para. [0107], “In the AR object 410, in advance, (X (obj), Y (obj)) is set as position (coordinate) information of a reference position 410P, and D (obj) is set as orientation information 410D”; para. [0190], “The AR object 410 that reflects the changes in shape, display color, etc. may be determined by AR object identification information”; para. [0212], “an added AR object comprised of a comment is added to a reality object comprised of an image of a real object existing in a real space.”). Note that: (1) an AR object as a comments affects a reality object of the environment in the target space; (2) the AR object’s coordinates are position information; and (3) the D (obj) is set as orientation information as an attribute as well as shape, display color, etc.
based on the target space data acquired by the acquisition circuit, the attribute and the position of the object in the target space, and the attribute and the position of the (para. [0230], “The object information display control unit 1114 generates a 3D model (reality object information) regarding the reality object 400 selected in the processing of S605, on the basis of the distance information about the reality object 400 acquired in the processing of S606 and the reality object image acquired by the out-camera 133 (S608). Incidentally, it is sufficient if the 3D model generated by this processing can be recognized as a position reference when adding a provisional added AR object such as a comment in the provisional object edit processing performed on the tablet terminal 200, and it is not necessary to accurately reproduce the shape of the reality object 400. Further, the reality object information includes the position information and orientation information regarding the reality object calculated in the processing of S607”). Note that: the 3D model(s) of object(s) with corresponding attributes and position information of the target space constitute the 3D target space model.
However, Yoshizawa fails to disclose, but in the same art of computer graphics, Becerra disclose
… a confidential target object …
… the target object …
the confidential target object is processed. (Becerra, Abstract, “in response to determining the first electronic document includes a confidential object, an alternative object associated with the confidential object, and generating a second electronic document by replacing the confidential object with the alternative object on the first electronic document”). Note that: (1) the confidential target object can be an object (e.g., a person) whose identification information as an attribute of an object needs to be protected in the same target space with other non-confidential objects; (2) the confidential target object has its own attribute and position like non-confidential objects, so that the target space model can be generated or constituted above; and (3) the confidential object can be replaced by an alternative non-confidential object with the corresponding attributes, which means that the confidential target object is processed.
Yoshizawa and Becerra are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply replacing a confidential object with alternative object, as taught by Becerra into Yoshizawa. The motivation would have been “in response to determining the first electronic document includes a confidential object, an alternative object associated with the confidential object, and generating a second electronic document by replacing the confidential object with the alternative object on the first electronic document” (Becerra, Abstract). The suggestion for doing so would allow to protect the confidential object by replacing the confidential object with an alternative non-confidential object. Therefore, it would have been obvious to combine Yoshizawa and Becerra.
Allowable Subject Matter
Claims 7-11, 14-15, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding dependent claim 7, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “an air conditioner or a ventilation device, and the processor is configured to recognize an air control port of the air conditioner or the ventilation device, and discriminate whether the air control port is a suction port or a blow-out port”.
Claims 8-11 depend from claim 7.
Regarding dependent claim 14, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “acquire piping system data representing a piping system of an air conditioner in the target space, and create the three-dimensional space model including thermal boundary conditions for the air conditioner based on the acquired piping system data”.
Claim 15 depends from claim 14.
Regarding dependent claim 17, in the context of claim as a whole, the prior art either alone or in combination does not teach or suggest the additional elements of: “acquire thermography data of the target space, outside air temperature data of the target space, solar radiation amount data, and thickness data of a fitting, and create a three-dimensional space model including a heat transmission coefficient of the fitting as a thermal boundary condition based on each piece of the acquired data”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhang et al. (Smart DC: An AI and Digital Twin-based Energy-Saving Solution for Data Centers, NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium) teaches an energy-saving solution based on Artificial Intelligence (AI) and digital twin in DC scenarios, called Smart DC. The proposed solution can reduce DCs' energy consumption by optimizing air distribution and reducing cooling redundancy.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights