Incident Notification: Kafka Based System Outage Thursday, Oct 27th, 2022
The Kafka-based event streaming platform becomes non-operational when a Kafka node goes offline due to exceeding capacity; the downstream impact of which is all Enlighted gateways connected to Energy Manager Cloud instances disconnect.
Our team has been able to mitigate the impact by applying a restart to the services, most gateways come back online, but there is a potential impact that a percentage do not. For those that do not, a power cycle will need to be performed, and if this does not resolve, the gateway will need to be replaced.
Who was Affected
- All customers using the Enlighted Energy Manager Cloud (EMC) Solution
Root Cause Analysis Discoveries
- A kafka bug has been discovered within the live instance which impacts the LogCleaner thread
- Allocated disk space was increased.
- Alert monitoring alert thresholds were lowered, and monitoring urgency was raised to improve response for Kafka platform environment issues.
- Urgently preparing minor Kafka upgrade to resolve bug
Current Actions: (10/27/2022)
- Support is working with customers affected by offline gateways
- Preparations are being made to perform the necessary upgrade
- Further communication will be posted with additional information
The FAQs are categorized into the following key topics based on functionality.
Installation, Communication, Connectivity, and Networking
What are the brand and server details of Enlighted Manage servers?
Manage EM-1500 (EM-2-02)
Management Appliance includes the license for 1000 sensors.
Allows the addition of licenses to support up to 1500 Sensors.
Lanner LEC-7230. Intel Celeron J1900 2.0 GHz, 64 GB SSD. Mounting bracket. Power adapter 120/240V.
Midrange EM-5000 (EM-2-03)
Management Appliance includes the license for 1000 sensors.
Allows the addition of licenses to support up to 5000 Sensors.
Lanner LEC-7070. 2 x Intel Core i7-3517UE, 1.7 GHz, 128 GB SSD. Mounting bracket. Power adapter 120/240V.
Enterprise Manage (EM-03-01)
Management Appliance includes the license for 5000 sensors. It allows the addition of licenses to manage up to 18000 sensors (10,000 sensors with BACnet points).
Dell PowerEdge R440 Rack Server. 4 x Intel Xeon Silver 4112, 2.6 GHz. 6 x 600 GB Hard Drives. Includes NEMA 5-15P to C13 Wall Plug, 125 Volt, 15 AMP, 10 Feet (3m), Power Cord, North America.
Which protocol standards does Manage support?
The following standards are supported: Ethernet/IP,DHCP,HTTPS,TLS 1.2,NTP, BACnet/IP
Which Internet of Things (IoT) communication protocols does Manage use?
As an edge device manager and data aggregator, the Manage server communicates using TLS connections with Enlighted Gateways.
Which wired or wireless building control system communication protocols does Manage use?
The Enlighted BACnet/IP provides integration between the Enlighted lighting system and any BACnet compatible Building Management System (BMS). The BMS with BACnet/IP capability can integrate, monitor, manage, and control the Enlighted lighting system through Group and Individual points, configured via BACnet. See Configuring BACnet Interface and refer to the BACnet Interface for Enlighted Systems-PICS.
Does any information enter or leave Manage via any other (including proprietary) protocols?
When cloud-connected, Manage sends data to the Enlighted Cloud via a TLS connection.
Does Manage support any naming convention/meta-tagging implementations?
Manage allows for sensor naming, but no data is kept on specific users and or occupants.
Does Manage process any confidential data (e.g., individually identifiable data - e.g., cameras, microphones, phones)
The Manage server does not process PII (Personally Identifiable Information).
What are the electrical characteristics of the Manage server device?
- 12 VDC power
- Standby Power 7.36 W
- Operational Power 11.43 W
Can Manage synchronize to an NTP server?
What wired network speeds does the Manage server support?
10/100/1000 Mbps Ethernet
Do I need a VMware license for the Enterprise Manage?
Yes. A free license to VMware vSphere Hypervisor 6.7 is available at my.vmware.com.
Can the on-prem Manage communicate to third party systems via both BACNET and APIs at the same time?
Yes, Manage can communicate.
What is Hard metering?
Hard (Hardware) metering is the energy data calculated using the driver and ballast in the fixture. Refer to the article Energy Metering with Enlighted Sensors.
What is Soft metering?
Soft (Software) metering is the energy data calculated by Manage based on the fixture's dimming level and profile. This requires a Power-Volt (PV) map to be uploaded to Manage. Most of the driver/ballast manufacturers provide the PV map. Refer to the article Setting up Soft Metering.
Can customers manage passwords and backup?
Yes, Facility administrators can govern Manage user access. Facility administrators should be able to download and restore the backup if needed.
Does Manage implement a network configuration/provisioning system?
Manage provides DHCP IP addresses to Enlighted Gateways. See Network and IT Design Guidance and Assigning Network IP Addresses (for on-premise Manage Server)
Does Manage have the capability to store/retain any historical/trend data? Describe what data is stored and how long it is retained on the device.
Data is retained for 45 Days on the local server. The data is also compressed and stored in the cloud for extended data availability.
Does Manage provide an open and documented application programming interface (API) and web services that can be consumed by third-party applications? Provide details on what elements of Manage are exposed and any functionality which is not via APIs.
What type of user interface does Manage provide for end-user interaction?
The Enlighted system provides a web GUI for administration and monitoring of the lighting system. The interface allows users to manage device and lighting configuration for the system. Additionally, it allows users to monitor device status and perform administrative functions like firmware updates, backups, etc. See Manage System Admin Guide.
Does Manage log all events?
Yes. Manage provides standard logs available in Linux as well as logs that are proprietary in our operating system.
Can I show the energy savings of your system on a TV in the lobby?
Yes. You can show the savings of your system on a TV in the lobby.
Why can't I integrate the Enlighted system with another lighting system through the UI?
As the Enlighted system is open, many partners choose to do such integrations with the Enlighted API. We haven't had any requests to do so via a User Interface as the API provides a more scalable and flexible integration solution. That said, Enlighted would be happy to work with 3rd parties that want to use our API to create a UI to facilitate integrations with other lighting systems.
Does your system work with an Astronomical clock?
Yes, this feature is supported in Gen 4 and Gen 5 sensors. See Understanding Profile Basic Attributes.
Does your system work with Circadian Rhythm?
Yes, this feature is supported in Gen 5 sensors controlling a Tunable White fixture. See Configuring Tunable White article.
If we do not have the final floor plan, can Enlighted change it later?
Yes. Contact Enlighted customer support. They provide this service.
How is the Energy Savings Report generated in Manage?
The Energy Graph shown below is an actual sample report for Energy Consumption. The data shows daily consumption by the hour and for longer periods, such as over the last seven days, 30 days, or monthly over the last year.
To save the Energy Consumption details to PDF or to export to CSV:
- Go to Manage> Facilities> Energy Consumption.
- Click Save as PDF to save the graph in PDF format and select the required location to save the file.
- Click Export to export the details to an Excel file (CSV) format and select the required location to save the file.
Refer to the article Viewing Energy Consumption for more information on Energy Consumption.
Do we have a Spec Sheet for ENCL-EM/FAN?
Yes. The part code ENCL-EM/FAN is an enclosure for Manage and PoE switch. This is a third-party box that we ship for installing Manage. Refer to the article Enclosure for Manage and PoE Switch.
Does Manage implement certificate/API key-based authentication to access the data through the web services API?
Yes. Manage implements API key-based authentication to access data via APIs. See User Authentication.
Does Manage support authentication through the exchange of keys during device provisioning and certificate-based authentication?
Yes. During the initial setup and cloud connection, a certification process has to be successfully configured.
Can the certificate authority credentials be revoked/rotated in the event of a compromise?
The Enlighted Security team can revoke the certification.
Does Manage support TLS encryption on a shared medium networking stack (e.g., Ethernet, WiFi)
Yes. Manage supports TLS 1.2.
How is authentication/access to configuration settings and firmware updates managed?
Only users with administrative roles can perform these actions.
Will *all* access credentials be provided and be fully configurable?
Access privileges and access control are in place and can be configured to give different users different security levels for access.
How are ‘engineering’ or ‘factory’ credentials managed?
All default credentials are changed per Enlighted best practices. Manage enforces a password change for the default administrative user.
Can Manage communicate with any external services (e.g., phone-home functionality or update services)
Manage can be configured to send data to the cloud. This has to be initiated by the user as this is not something that it does automatically.
If Manage supports communication with external servers, can this be fully disabled?
Yes. Remote access is needed to enable additional application support but can be disabled at any time. Manage can maintain an optional diagnostics connection via an SSH connection from Manage outbound the Enlighted Cloud. This connection can be disabled or restricted by firewalls with no effect on core system functionality.
Does Manage have any connectivity options available that may bypass the network firewall (e.g., tunneling/VPN options, 3G/4G cellular connectivity)
Yes. 4G modems and tunneling are possible. The use of 4G is at the customer's discretion. Tunneling is used for an optional diagnostic connection. The connection is recommended to allow for remote support but can be disabled or enabled as needed.
If Manage uses non-IP wireless technology how is information exchanged between the network interface (wired or wireless) from the non-IP wireless communications?
This communication is done at layer 2. The traffic communicates with the sensors across 802.15.4 then through the Enlighted Gateway at which it is converted to Wired Ethernet.
Quick Daylight Response FAQ
By default, the fixture adjusts its light during the daylight harvesting process slowly so that it does not really attract occupants' attention. Is there a way that ambient light (daylight) influences the light level of the fixture immediately and very quickly (within one or two seconds)?
For individual daylighting, it can be quite fast (seconds).
For daylight groups, it occurs more slowly (over minutes).
Generally, the lights will brighten faster than they will dim. In some cases with motion and complete darkness, they may brighten immediately.
The dimming process also can take up to 30s or a minute at an individual fixture. Is there a feature in EM to enable a more ‘real-time’ reaction?
This can be set in the advanced profile setting for individual daylight harvesting (not daylight groups). The two settings shown below work in tandem, i.e., if you change the highlighted value from default 10 seconds to 1 second, it can be quite disconcerting to occupants as every slight change in light will affect the light levels and then, the occupants will be stuck with the change for the next 10 minutes. If you change both values, the cycling of light can be quite annoying to the users. The time period of 10 seconds and 10 minutes is recommended. Customers may think they want a fast response to daylight, but actually, Enlighted differentiates by ensuring consistent light changes before making a change. A delay ensures a prolonged light change rather than having a fast response time.
Perhaps 5 seconds and 5 minutes is a good compromise for more aggressive response time.
To change the settings for a fixture, select a fixture from the floor plan and double-click to view the Fixture Details window. Select the Profile tab and scroll down to view the Advanced menu.
Demand Response FAQ
Can I adjust a Demand Response (DR) Override lighting profile during a Demand Response event and have the changes take effect within 5 minutes.
Yes. When a scheduled DR event is active, you can change the Min and Max Light levels and other profile parameters in the override profile of the fixture template. The newly updated override values may take up to five minutes to act on the override event.
To modify the values of a fixture profile when a schedule DR is active, navigate to the Profile Templates page in the left panel, and click Edit to select the fixture profile template. Scroll down to the Override Profiles section and change the lighting values.
Adobe Flash Support FAQ
Will my lights stop working if I don’t update my Enlighted System before 1/1/21?
No, your lights will continue to work as previously configured. Lighting functionality does not rely on Flash.
Which sections of the Energy Manager UI currently use Flash?
The major elements of the Energy Manager that rely on are Flash are the dashboard, floor plan views (includes floorplan-based commissioning and configuration), fixture outage report, gateway outage report, and usage by profile report.
What are my options if I choose not to update the Energy Manager but want to continue to see the Flash portions of the Energy Manager UI?
Given that Adobe and all major browsers will no longer support Flash, security or performance issues will not be addressed should they arise. Enlighted advises against this course of action. However, it is possible to run legacy versions of browsers, but you will be doing this at your own risk. If you choose to update your application later, you will be able to do so as the application update process is not affected by Flash.
Do I need to buy anything to upgrade to Manage 4.0?
No, customers do not need to purchase the Manage 4.0 update. If your system is not compatible with Manage 4.0 for some reason, you may need to purchase newer server hardware or migrate to a hosted system.
Is my Energy Manager server compatible with Manage 4.0?
If you purchased your energy Manager prior to 2016, you are currently running Energy Manager version 3.5.5 or older, or your Enlighted Server looks like the below image, your on-prem Energy Manager might be incompatible with Manage 4.0.
Please contact Enlighted support or your account representative for further help. If you have deployed the Energy Manager application on a self-managed virtual machine (VEEM), please contact Enlighted Support to confirm compatibility. Otherwise, your system hardware is compatible with Manage 4.0.
Can I perform the update myself?
Yes! Instructions on how to update the application can be found on our support site HERE. Hosted applications are always updated by Enlighted support in coordination with the customer. Please contact Enlighted support if you have an Energy Manager in the Cloud (EMC) and would like to be updated to Manage 4.0.
Do I need to have physical access to the Energy Manager server to perform the update?
No. Only network connectivity to the application is required to perform the update an on-premise or customer-managed virtual machines. If the Energy Manager is not accessible over the internet, then remote access can be achieved in multiple ways; some examples below:
- Remote access via Team Viewer/Remote Desktop (requires an internet-connected computer on the same network as the Energy Manager)
Astronomical Clock FAQ
Will the Astroclock on/off times adjust every day?
Yes. Assuming sunset and sunrise time change more than a minute each day, and the sensors communicate with Manage, the time will change each day. Though, there are tons of other settings in groups and profiles that could impediment the change. So, it depends on the profile and group settings.
Do the sensors need to “sync” with Manage to adjust the on/off times?
Yes. Because the sensor needs to sync only once during the 24 hour period, it is not required for the sensors to sync at sunrise and sunset.
What time is used for the Astronomical clock? Is it the sunset, twilight, or last light?
Yes. It should change considering the other profiles, and group settings do not override the change. It is exactly at the sunset and sunrise based on the latitude/longitude available in the facility settings (use the plus or minus 'x' minutes if you want twilight or last light. Refer to the article Understanding Light Profiles.
RMA Warranty FAQ
How do we order parts for the RMA warranty?
All Enlighted hardware products come with a limited five-year Return Merchandise Authorization (RMA) warranty period from the date of shipment or based on a valid product under warranty based on the terms and conditions of your contract agreement. If Enlighted receives an RMA within the warranty period, Enlighted will replace the product with a new or refurbished part or product.
The following conditions must be met to qualify for an RMA:
- The product must be within the applicable warranty period.
- The defective products will be accepted only if they are not physically broken or damaged.
- The RMA number must be included along with the returned product.
- If requesting a full refund for newly purchased devices in the original condition, the product must be returned to Enlighted within 60 days from the date of shipment.
Send an email to email@example.com to submit the following information:
- Purchase Order number
- Project Name or Site Address
- Contact Name, Address, and Phone number
- A brief description of the issue
- Product Number and Quantity
The Enlighted Customer Support team will provide you with an RMA number and email the instructions for returning the parts along with a shipping label or an Enlighted FedEx account number to return the product.
What are the terms of contract conditions and warranty?
Please refer to the article Terms and Conditions to understand the warranty information.