We’ve noticed a gap in the market for an affordable, production ready VTN. To meet demand, we’re offering a yearly license in exchange for regular updates and support for the EPRI VTN. The updates will address issues related to running the VTN in a production like environment aimed at pilots using thousands of VENs. Space in the program is limited so contact us soon!
Issues with the EPRI VTN
OpenADR has been gaining favor as the goto demand response protocol for programs throughout the US with Europe and Japan also showing strong interest. California is providing an especially strong push for demand response and distributed energy resources in general with the EPIC Program and longer term pilots such as the Southern California Edison Charge Ready Program requiring a 10 year commitment. Smaller utilities, Municipals, Coops and other companies interested in demand response are doing their own testing with freely available and low cost tools such as the EPRI VTN, the Quality Logic cloud based VTN, or the nebland software virtual machine appliance. Customers are starting to ask the question: how do I move from lab testing to pilots using these tools?
The short answer is, in its current form, the EPRI VTN is not well suited for day to day operations for a long(er) term pilot connected to many VENs. But we’re changing that. Here are some issues that need to be addressed:
- High availability installation instructions and best practices
- Performance issues apparent when dealing with thousands of devices
- Improve the API feature and provide detailed documentation for integrating with 3rd party systems
- Compliance testing
We’ll address these issues and make the changes available to licensed customers through regularly scheduled releases. We’re opening enrollment to 5 companies but only 2 spots remain. Depending on feedback, we’ll open up enrollment to a few more customers starting in January of 2018 on a first come, first serve basis. Use the contact form at the bottom of this page to claim your spot.
Read on for details of the issues and how we intend to address them.
When delivered over http, an OpenADR VTN can install and scale like any other web application. This basically means an http load balancer in front of multiple front end instances and a scalable database in the back. We will test different setups and collect performance data with the nebland software VTN load testing system, and provide instructions and recommendations for performing a high availability installation and handling day to day operations.
Performance issues we’ve seen fall into the following categories: inefficient event generation, java singleton issue, torquebox cpu and memory consumption, interval data storage and retrieval.
Inefficient Event Generation
When generating an event, the VTN does a blocking call and generates an event for every VEN target. This means the user will be stuck with a non-responsive UI waiting for event generation to complete.
In many situations, the event payload is exactly the same for every VEN target so there’s no need to generate the same message multiple times. When targeting a small amount of VENs, this isn’t a problem, but as the number of targeted VENs increases, so does the time needed to generate the message. This provides a bad user experience at best, and can simply timeout and fail at worst, giving the user an unfriendly error message. Two changes need to be made: 1) Move event generation to a background processor, and 2) If no VEN specific information such as VEN ID or Party ID are located in the event, the event message will only be generated once.
A Java singleton is used throughout the code for converting between Java POJOs that can be converted to and from XML, and ruby active record objects. This interaction between Java and Jruby is very resource intensive. The singleton object with locks was required because the object isn’t threadsafe, and there’s an enormous performance hit when creating the object. The locked signleton objects means only one VEN can receive data at a time – no matter how many threads are available.
The solution: move everything related to message handling to a separate java process. This will immediately improve performance by avoiding the interaction between Java and Jruby, and this model is easier to scale. Separating the code will make it easier to refactor the Java code to take advantage of multiple threads.
Moving the Java code out of the core app has additional benefits. Using Java in Jruby has proven to be very cumbersome due to the complexity of Java. Separating the code will make both the Rails code Java code easier to reason about, and the two components can be tested independently.
Torquebox Resource Usage
Torquebox has been a pain point for a number of reasons: it’s slow to start, it uses lots of CPU and memory, and most importantly: support seems to be fading. The bottom line is: torquebox must go. Resolution of the previous issue will create a separation between the Java code and Rails code which means we will be able to switch to standard ruby and use one of the more popular Rails application servers.
One of the larger obstacles to making the system more performant is handling interval data both in storage and retrieval. Instead of a traditional RDBMS, we’ll explore using a time series database such as InfluxDB which is much better suited for handling this type of data. Installations that don’t use a lot of interval data should work just fine with a traditional RDBMS, however, so the use of a time series database should be optional.
The Rails code has performance issues related to converting many database records into Ruby objects (e.g. when generating charts). We’ll explore options to improve this performance.
Improved API Feature
The VTN includes an extensible API feature for integration with 3rd party systems. Documentation for this feature is basically non-existent, there are no concrete examples demonstrating how to use the feature, and it needs authentication and authorization. We’ll address all of these issues.
The first release of the EPRI VTN was compliance tested but subsequent releases have not been. The latest release should still be compliant, but there’s no guarantee. We will use our automated instance of the Quality Logic testharness to integrate running the compliance tests with our release process, and guarantee that each release passes all compliance tests.
What we’re offering
In exchange for a yearly license fee, we’re offering access to the above improvements as they’re implemented as well as a few hours of support to help you manage your instances.
This project won’t happen overnight. We’re planning this work to happen over the next two years, but if there’s enough interest, we’ll accelerate the time line. Space is limited so don’t hesitate to contact us!